Jorge Middleton (Customer) asked a question.
I'm evaluating DataDirect to upload binary files i.e. images, pdf, word document into Salesforce from AWS Glue. The object or table used for this is ContentVersion (Salesforce object) and the column is VersionData.
In the logs I can see that VersionData is processed as BINARY but I don't know how to pass a value from AWS Glue DataFrame or DynamicDataFrame.
spy(2021-10-01T03:54:09.524)>> ResultSetMetaData[1].getColumnLabel(int column)
spy(2021-10-01T03:54:09.524)>> column = 30
spy(2021-10-01T03:54:09.524)>> OK (VERSIONDATA)
spy(2021-10-01T03:54:09.524)>> ResultSetMetaData[1].getColumnType(int column)
spy(2021-10-01T03:54:09.524)>> column = 30
spy(2021-10-01T03:54:09.524)>> OK (-4)
spy(2021-10-01T03:54:09.524)>> ResultSetMetaData[1].getColumnTypeName(int column)
spy(2021-10-01T03:54:09.524)>> column = 30
spy(2021-10-01T03:54:09.524)>> OK (BINARY)
spy(2021-10-01T03:54:09.524)>> ResultSetMetaData[1].getPrecision(int column)
spy(2021-10-01T03:54:09.524)>> column = 30
spy(2021-10-01T03:54:09.524)>> OK (5242880)
I can see an error in AWS Glue logs but not sure if that is associated with this:
Uploading file: files/year=2020/month=01/day=03/apache-maven-3.6.0-bin.tar.gz
2021-10-01 03:54:42,664 WARN [Executor task launch worker for task 1.1 in stage 8.0 (TID 24)] partner.PartnerJDBCRecordWriter (PartnerJDBCDataSink.scala:partitionAbort(192)): Glue ETL Marketplace: JDBC Data Sink: write partition failed, abort writer.
2021-10-01 03:54:42,665 ERROR [Executor task launch worker for task 1.1 in stage 8.0 (TID 24)] executor.Executor (Logging.scala:logError(94)): Exception in task 1.1 in stage 8.0 (TID 24)
com.ddcloud.jdbc.sforcebase.c: [DD Cloud][SForce JDBC Driver]The hexadecimal string is invalid.
at com.ddcloud.jdbcspysforce.SpyLogger.sqlException(Unknown Source)
at com.ddcloud.jdbcspysforce.SpyStatement.executeBatch(Unknown Source)
at com.amazonaws.services.glue.marketplace.partner.PartnerJDBCRecordWriter.writePartition(PartnerJDBCDataSink.scala:173)
at com.amazonaws.services.glue.marketplace.connector.GlueCustomDataSink.$anonfun$defaultWriteDynamicFrame$1(CustomDataSink.scala:83)
at com.amazonaws.services.glue.marketplace.connector.GlueCustomDataSink.$anonfun$defaultWriteDynamicFrame$1$adapted(CustomDataSink.scala:72)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-10-01 03:54:42,672 INFO [dispatcher-Executor] executor.CoarseGrainedExecutorBackend (Logging.scala:logInfo(57)): Got assigned task 25
Hi Jorge,
If you have a column which can store images or documents or audio/video files , then we can send binary data. VersionData is Base64 column for which Binary data is applicable.
Sample Code
FileInputStream fin = new FileInputStream("javafx_logo.jpg");
pstmt.setBinaryStream(3, fin);
Let me know if you have any questions.
Sai
Hi Sai,
Thanks for your reply.
In AWS Glue we don't have the capability to set the prepared statement. AWS Glue manages dataset of information like tables in SQL that you transform, merge, aggregate, etc. One of the dataset has a base64 column which is then use to create VersionData table.
Do you know if I can use the base64 representation of a file? In my test it didn't work and I got the error "[DD Cloud][SForce JDBC Driver]The hexadecimal string is invalid."
Regards,
Jorge