Click here to Skip to main content
15,886,756 members
Home / Discussions / Cloud Computing
   

Cloud Computing

 
QuestionOffice 365 + IMAP + OAuth2 [Solved] Pin
Richard Deeming15-Apr-20 4:03
mveRichard Deeming15-Apr-20 4:03 
AnswerRe: Office 365 + IMAP + OAuth2 Pin
Richard Deeming20-Apr-20 0:41
mveRichard Deeming20-Apr-20 0:41 
AnswerRe: Office 365 + IMAP + OAuth2 Pin
Richard Deeming28-Apr-20 0:21
mveRichard Deeming28-Apr-20 0:21 
GeneralRe: Office 365 + IMAP + OAuth2 [Solved] Pin
OriginalGriff6-Feb-24 21:18
mveOriginalGriff6-Feb-24 21:18 
GeneralRe: Office 365 + IMAP + OAuth2 [Solved] Pin
Richard Deeming6-Feb-24 21:20
mveRichard Deeming6-Feb-24 21:20 
QuestionASP.NET + Multi Form using Textbox problem Pin
kerek224-Feb-20 21:42
kerek224-Feb-20 21:42 
AnswerRe: ASP.NET + Multi Form using Textbox problem Pin
Richard MacCutchan24-Feb-20 23:17
mveRichard MacCutchan24-Feb-20 23:17 
QuestionAzure Data Factory pipeline into compressed Parquet file: “java.lang.OutOfMemoryError:Java heap space” Pin
VictorSotnikov16-Sep-19 10:20
VictorSotnikov16-Sep-19 10:20 
I have met the following problem. I have a pipeline that reads data from an MS SQL Server and stores them into a file in a BLOB container in Azure Storage. The file has Parquet (or Apache Parquet, as it is also called) format.
So, when the “sink” (output) file is stored in a compressed way (snappy, or gzip – does not matter) AND the file is large enough (more than 50 Mb), the pipeline failed. The message was the following:
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space\ntotal entry:11\r\njava.util.ArrayDeque.doubleCapacity(Unknown Source)\r\njava.util.ArrayDeque.addFirst(Unknown Source)\r\njava.util.ArrayDeque.push(Unknown Source)\r\norg.apache.parquet.io.ValidatingRecordConsumer.endField(ValidatingRecordConsumer.java:108)\r\norg.apache.parquet.example.data.GroupWriter.writeGroup(GroupWriter.java:58)\r\norg.apache.parquet.example.data.GroupWriter.write(GroupWriter.java:37)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:87)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:37)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)\r\norg.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:292)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchWriter.addRows(ParquetBatchWriter.java:60)\r\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Work_Work"
}

The "Work_Work" is name of a Copy Data activity in the pipeline.
If I turn the compression off (the generated BLOB file is uncompressed), the error does not happen.
Is this the error described in https://docs.microsoft.com/en-us/azure/data-factory/format-parquet: the “…If you copy data to/from Parquet format using Self-hosted Integration Runtime and hit error saying "An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space", you can add an environment variable _JAVA_OPTIONS in the machine that hosts the Self-hosted IR to adjust the min/max heap size for JVM to empower such copy, then rerun the pipeline….”?

If it is, have I understood correctly that I have to do the following:
To go to a server where the “Self-hosted Integration Runtime” (still have no idea what it is) and increase the max heap size for JVM. Is this correct?
If it is, my next question is: how large the max heap size should be? My pipeline can generate a file whose size will be 30 GB.
What “max heap size” can guarantee that such a file will not cause the fail?
QuestionHow to search for a word or color on a website? Pin
l@rs_30-Aug-19 8:17
l@rs_30-Aug-19 8:17 
AnswerRe: How to search for a word or color on a website? Pin
Afzaal Ahmad Zeeshan30-Aug-19 12:02
professionalAfzaal Ahmad Zeeshan30-Aug-19 12:02 
AnswerRe: How to search for a word or color on a website? Pin
Member 1502871723-Dec-20 1:45
Member 1502871723-Dec-20 1:45 
AnswerRe: How to search for a word or color on a website? Pin
Peter Groft6-Jul-22 3:10
Peter Groft6-Jul-22 3:10 
QuestionWhat are some of Azure costing to get minimal app running? Pin
maze328-May-19 5:11
professionalmaze328-May-19 5:11 
QuestionHow to see the payload sent to my Azure web service Pin
Hypermommy2-May-18 2:57
Hypermommy2-May-18 2:57 
AnswerRe: How to see the payload sent to my Azure web service Pin
Richard Deeming2-May-18 8:45
mveRichard Deeming2-May-18 8:45 
AnswerRe: How to see the payload sent to my Azure web service Pin
Afzaal Ahmad Zeeshan6-May-18 3:26
professionalAfzaal Ahmad Zeeshan6-May-18 3:26 
GeneralRe: How to see the payload sent to my Azure web service Pin
Hypermommy11-Jul-18 4:20
Hypermommy11-Jul-18 4:20 
QuestionAccessing Graph.microsoft.com on behalf of a customer tennant Pin
me@dagsunde.com26-Feb-18 0:48
professionalme@dagsunde.com26-Feb-18 0:48 
AnswerRe: Accessing Graph.microsoft.com on behalf of a customer tennant Pin
Afzaal Ahmad Zeeshan6-May-18 3:15
professionalAfzaal Ahmad Zeeshan6-May-18 3:15 
QuestionWindows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
CMS196821-Jan-17 0:45
CMS196821-Jan-17 0:45 
AnswerRe: Windows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
Afzaal Ahmad Zeeshan21-Jan-17 2:13
professionalAfzaal Ahmad Zeeshan21-Jan-17 2:13 
GeneralRe: Windows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
CMS196823-Jan-17 9:31
CMS196823-Jan-17 9:31 
QuestionRead Azure Filesystem-Files (not VM) Pin
gpc4419-Jun-16 23:44
gpc4419-Jun-16 23:44 
AnswerRe: Read Azure Filesystem-Files (not VM) Pin
debasish mishra7-Jan-18 20:31
professionaldebasish mishra7-Jan-18 20:31 
QuestionAzure subdomains... Pin
R. Giskard Reventlov8-May-16 14:18
R. Giskard Reventlov8-May-16 14:18 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.