Spark Connector for NetSuite is a SOAP web service wrapper around the NetSuite web service published here.

This library requires Spark 2.x+

For Spark 1.x support, please check spark1.x branch.

You can link against this library in your program at the following ways:

<dependency>
    <groupId>com.springml</groupId>
    <artifactId>spark-netsuite_2.11</artifactId>
    <version>1.1.0</version>
</dependency>

libraryDependencies += "com.springml" % "spark-netsuite_2.11" % "1.1.0"

This package can be added to Spark using the --packages command line option. For example, to include it when starting the spark shell:

$ bin/spark-shell --packages com.springml:spark-netsuite_2.11:1.1.0

• Construct Spark Dataframe using NetSuite data - User has to provide NetSuite web service request and list of XPath to read data from NetSuite. The XPath will be evaluated against NetSuite web service response and dataframe will be constructed based on that. Records will be searched based on the user provided request and further records will be fetched using searchMoreWithId

• email: NetSuite account user Id
• password: NetSuite account passsword
• account: NetSuite account Id
• applicationId: NetSuite application Id
• role: (Optional) NetSuite Role Id. Default value is 3
• pageSize: (Optional) Number of records to pulled in a single request. Max pageSze is 1000. Default value is 100.
• request: NetSuite Web Service search request. This request will be used to search for records from NetSuite. Sample request is present over here
• recordTagPath: (Optional) XPath of the response element which should be considered as record. Default value is //platformCore:record
• xpathMap: Location of CSV file which should contain fieldName and its XPath. Sample file is present over here
• namespacePrefixMap: Location of CSV file which should contain prefix and its corresponding namespace. Sample file is present over here
• schema: (Optional) Schema to be used for constructing dataframes. If not provided all fields will be of type String

import org.apache.spark.sql.SQLContext

// Construct Dataframe from NetSuite records
// Search request to be executed against NetSuite Web Service
// Here Customers are fetched
val request = """
<search xmlns="urn:messages_2016_1.platform.webservices.netsuite.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <searchRecord xmlns:ns7="urn:relationships_2016_1.lists.webservices.netsuite.com" xsi:type="ns7:CustomerSearch">
        <ns7:basic xmlns:ns8="urn:common_2016_1.platform.webservices.netsuite.com" xsi:type="ns8:CustomerSearchBasic"></ns7:basic>
    </searchRecord>
</search>"""

// Below constructs dataframe by executing search and searchMoreWithId operations 
val df = spark.read.
    format("com.springml.spark.netsuite").
    option("email", "netsuite_email").
    option("password", "netsuite_password").
    option("account", "netsuite_account").
    option("applicationId", "netsuite_application_id").
    option("request", request).
    option("xpathMap","/home/xpath.csv").
    option("namespacePrefixMap","/home/namespaces.csv").
    load() 

# Search request to be executed against NetSuite Web Service
# Here Customers are fetched
netsuite_request <- "<search xmlns=\"urn:messages_2016_1.platform.webservices.netsuite.com\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><searchRecord xmlns:ns7=\"urn:relationships_2016_1.lists.webservices.netsuite.com\" xsi:type=\"ns7:CustomerSearch\"><ns7:basic xmlns:ns8=\"urn:common_2016_1.platform.webservices.netsuite.com\" xsi:type=\"ns8:CustomerSearchBasic\"></ns7:basic></searchRecord></search>"

// Below constructs dataframe by executing search and searchMoreWithId operations 
df <- read.df(source="com.springml.spark.netsuite",
      email="netsuite_email",
      password="netsuite_password",
      account="netsuite_account",
      applicationId="netsuite_application_id",
      role="netsuite_user_role",
      pageSize="page_size",
      request=netsuite_request,
      recordTagPath="//platformCore:record",
      xpathMap="/home/xpath.csv",
      namespacePrefixMap="/home/namespace.csv")

This library is built with SBT, which is automatically downloaded by the included shell script. To build a JAR file simply run sbt/sbt package from the project root.