Talend Useful Links

1) 100 - Best Videos
http://meta-guide.com/videography/100-best-talend-videos/

2) DataBase Connection
https://www.youtube.com/watch?v=FezmHmYm-4c

3) MongoDB Connection
https://www.youtube.com/watch?v=zRoIiLlcnQE
https://www.talendforge.org/forum/viewtopic.php?id=27589

4) dynamic schema job
http://bekwam.blogspot.sg/2011/06/dynamic-schemas-in-talend-open-studio.html
https://www.youtube.com/watch?v=bLq8ip6nH5g&gl=SG&hl=en-GB

5) compare two tables via query
http://anilmca4u.blogspot.sg/2012/12/how-to-compare-two-tables-for.html

6) Insert into MongoDB 
https://www.youtube.com/watch?v=BX9ncCCmN8Y
https://satishchennupati.wordpress.com/2015/01/20/mongo-db-operations-using-talend/ (tMap)

7) MongoDB to postgrey sql
http://copyrightdev.tumblr.com/post/109311096968/using-talend-big-data-to-move-data-from-mongodb-to

8) Converting DataType 
http://talendhunter.blogspot.sg/2013/06/tutorial-converting-data-types-with.html

9) Pass flow data from Child to Parent Job in Talend
https://www.youtube.com/watch?v=msk3HedymMI

10) Pass parameters and variables to child jobs in Talend Open Studio
https://www.youtube.com/watch?v=D6BrfnFKSLQ
http://bekwam.blogspot.sg/2011/05/passing-parameters-and-variables-to.html

11) Comparing data in an RDBMS to data in MongoDB (Sonar Diff)
https://www.youtube.com/watch?v=REy6oT9bSw8

12) Add Hyperlink to Cells - Excel Java POI Example Program
http://thinktibits.blogspot.sg/2012/12/Java-POI-Create-Hyperlink-Excel-Cell-Example-Program.html

13) create Bar Chart in Excel - Java POI Example Program
http://thinktibits.blogspot.sg/2012/12/Create-Bar-Chart-Excel-Java-Example-Program-POI-JFreeChart-Tutorial.html
https://help.talend.com/display/TalendOpenStudioComponentsReferenceGuide55EN/tBarChart

14) Combining Excel Spreadsheets with tUnite in Talend Open Studio
http://talendhunter.blogspot.sg/2013/06/combining-excel-spreadsheets-with.html

15) Error Handling
http://bekwam.blogspot.sg/2011/04/three-error-handling-strategies-in.html

16) Parametrized Queries for MongoDB in Jaspersoft
https://www.youtube.com/watch?v=fVeYyLCLph4

17) Talend - Cluster
https://help.talend.com/display/KB/How+to+create+a+clustered+server+in+Talend+Administration+Center+-+Talend+v5.2+features
https://help.talend.com/display/TalendAdministrationCenterUserGuide55EN/6.3+Deploying+a+Job+or+a+Route+in+cluster+mode

http://bekwam.blogspot.sg/2011/04/three-error-handling-strategies-in.html

18) Pass context parameters via command line
http://www.vikramtakkar.com/2013/04/how-to-pass-context-parameters-to.html

19) How to pass Context parameters to Talend Job through command line
http://www.vikramtakkar.com/2013/04/how-to-pass-context-parameters-to.html

20) Mongodb bulk Load/Read
https://www.youtube.com/watch?v=QTzVetSm28s
http://talendhunter.blogspot.sg/2013/06/handling-empty-json-object-in-talend.html

21) Reading MongoDB Subdocuments
http://community.jaspersoft.com/blog/dealing-nested-documents-mongodb-and-talend-aka-baking-cake
http://youtu.be/82p_FnAPGb4
https://help.talend.com/display/TalendComponentsReferenceGuide54EN/tMongoDBOutput

22) Handling JSON
http://bekwam.blogspot.sg/2011/06/handling-empty-json-object-in-talend.html
https://www.talendforge.org/forum/viewtopic.php?id=25943

23) Write Document as Array
https://www.talendforge.org/forum/viewtopic.php?id=44448

Spoon CRASH ERROR java: cairo-misc.c:380: _cairo_operator_bounded_by_source

Spoon CRASH ERROR "java: cairo-misc.c:380: _cairo_operator_bounded_by_source


Solution : 

Add the below line in the spoon.sh OPT variable will resolve the problem

"-Dorg.eclipse.swt.internal.gtk.cairoGraphics=false"

Step by Step Hive User Defined functions (UDF)

Example simple UDF function  (StringUtilsUDF.java)

Step 1  :  Wrote simple Java function -  example (concat first name & lastName ) – which can be done via hive built in function
Step 2 : ADD JAR /home/gse/stringHiveUDF-1.0.jar;
Step 3 : CREATE TEMPORARY FUNCTION stringcat as 'com.test.udfs.StringUtilsUDF';
Step 4 :  Use the function in the hive select query            
                                select stringcat(billing_analyst_fname,billing_analyst_lname) from accounts


where account_number = 133708;
OK
Naoki,Ando
Time taken: 0.135 seconds, Fetched: 1 row(s)



StringUtilsUDF.java

package com.test.udfs;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;

public class StringUtilsUDF extends UDF {
private Text result = new Text();
  public Text evaluate(Text strFirst, Text strLast) {
 if (strFirst != null && strLast != null) {
 result.set(StringUtils.strip(strFirst.toString()) + "," + StringUtils.strip(strLast.toString()));
} else {
if (strFirst != null) {
result.set(StringUtils.strip(strFirst.toString()));
} else if (strLast != null) {
result.set(StringUtils.strip(strLast.toString()));
}else{
return null;
}
}
 return result;
    }
}

invalid LOC header (bad signature)

 some binary file being broken, most likely one of the dependencies.  just delete the respective maven repository and build again
rm -rf ~/.m2/repository/

example
       rm -rf ~/.m2/repository/org

Pagination in MongoDB - can be achieved via skip and limit

Pagination in MongoDB - can be achieved via skip and limit

     skip - NUMBER_OF_ITEMS * (PAGE_NUMBER - 1)
.limit(NUMBER_OF_ITEMS )

MongoDB aggregation result exceeds maximum document size (16MB)

MongoDB aggregation result exceeds maximum document size (16MB)
 
{
    "errmsg" : "exception: aggregation result exceeds maximum document size (16MB)",
    "code" : 16389,
    "ok" : 0
}

Solution : allowDiskUse to true or limit critera to max elements


Example

AggregationOptions aggregationOptions = new AggregationOptions(true,false,null);
Aggregation aggregation = newAggregation(
match(criteria),
limit(10),
sort(Sort.Direction.ASC,  "OrderSubmissionDate")
).withOptions(aggregationOptions);

In mongoDB - newAggregation throws match compilation error

Aggregation aggregation = newAggregation (
match(criteria)
)

solution
Import static newAggregation instead of nonstatic newAggregation
import static org.springframework.data.mongodb.core.aggregation.Aggregation.newAggregation;