In order to export data from LinkedIn to Google BigQuery using Node.js, it is necessary to utilize both the LinkedIn API and the BigQuery API. This process can be broken down into the following high-level steps: first, register as a developer on the LinkedIn API platform and obtain an access token, which will allow for the authentication of requests made to the LinkedIn API and the retrieval of data from your LinkedIn account or a public LinkedIn account. Next, use the BigQuery API to create a new dataset and table within your BigQuery project, into which the data from LinkedIn can be loaded. To make use of the LinkedIn and BigQuery APIs, it will be necessary to install the required packages in your Node.js environment; for LinkedIn, the linkedin-sdk package can be utilized, and for BigQuery, the @google-cloud/bigquery package is recommended. Using the Node.js request module or a similar package, make HTTP requests to the LinkedIn API in order to retrieve the desired data, and then use the @google-cloud/bigquery package to authenticate requests to the BigQuery API and load the data into the previously created BigQuery table. Once the data is in BigQuery, it can be analyzed and manipulated as needed using SQL queries.
First, you’ll need to register as a developer on the LinkedIn API platform and obtain an access token. You can use this access token to authenticate your requests to the LinkedIn API and retrieve data from your LinkedIn account or a public LinkedIn account.
Once you have the data you want to export from LinkedIn, you can use the BigQuery API to create a new dataset and table in your BigQuery project. You can then use the API to load the data from LinkedIn into the table.
To use the LinkedIn and BigQuery APIs, you’ll need to install the necessary packages in your Node.js environment. For the LinkedIn API, you can use the linkedin-sdk package. For the BigQuery API, you can use the @google-cloud/bigquery package.
You can use the Node.js request module or a similar package to make HTTP requests to the LinkedIn API and retrieve the data you want to export. You can then use the @google-cloud/bigquery package to authenticate your requests to the BigQuery API and load the data into your BigQuery table.
Once you have the data in BigQuery, you can use SQL queries to analyze and manipulate the data as needed.
Here is an example of how you could use the linkedin-sdk and @google-cloud/bigquery packages to export data from LinkedIn to Google BigQuery in Node.js:
const LinkedIn = require('linkedin-sdk');
const {BigQuery} = require('@google-cloud/bigquery');
async function exportData() {
// Replace these values with your own
const clientId = 'your_client_id';
const clientSecret = 'your_client_secret';
const accessToken = 'your_access_token';
const projectId = 'your_project_id';
const datasetId = 'your_dataset_id';
const tableId = 'your_table_id';
// Authenticate to LinkedIn and retrieve data
const linkedin = new LinkedIn(clientId, clientSecret);
linkedin.setAccessToken(accessToken);
const data = await linkedin.people.asMember('~:(id,first-name,last-name)');
// Initialize the BigQuery client
const bigquery = new BigQuery({
projectId: projectId
});
// Load the data into a BigQuery table
const options = {
schema: 'id:string,first_name:string,last_name:string',
createDisposition: 'CREATE_IF_NEEDED',
writeDisposition: 'WRITE_APPEND',
};
const [job] = await bigquery
.dataset(datasetId)
.table(tableId)
.load(data, options);
console.log(`Job ${job.id} completed.`);
}
exportData();
This code authenticates to LinkedIn using the linkedin-sdk package and retrieves data from the user’s profile. It then uses the @google-cloud/bigquery package to create a new table in a BigQuery dataset and load the data into the table.
Keep in mind that you’ll need to replace the placeholder values in the code with your own LinkedIn client ID, client secret, access token, and BigQuery project, dataset, and table IDs.
You’ll also need to ensure that you have the necessary packages installed and that you have set up authorization for the BigQuery API.
Are you eager to start sending Instagram data to Google Big Query using Node.js and have not found snippets of code needed to connect the dots?
First, you’ll need to register as a developer on the Instagram API platform and obtain an access token. You can use this access token to authenticate your requests to the Instagram API and retrieve data from your Instagram account or a public Instagram account.
Once you have the data you want to export from Instagram, you can use the BigQuery API to create a new dataset and table in your BigQuery project. You can then use the API to load the data from Instagram into the table.
To use the Instagram and BigQuery APIs, you’ll need to install the necessary packages in your Node.js environment. For the Instagram API, you can use the instagram-private-api package. For the BigQuery API, you can use the @google-cloud/bigquery package.
You can use the Node.js request module or a similar package to make HTTP requests to the Instagram API and retrieve the data you want to export. You can then use the @google-cloud/bigquery package to authenticate your requests to the BigQuery API and load the data into your BigQuery table.
Once you have the data in BigQuery, you can use SQL queries to analyze and manipulate the data as needed.
Here is an example of how you could use the instagram-private-api and @google-cloud/bigquery packages to export data from Instagram to Google BigQuery in Node.js:
const InstagramPrivateAPI = require('instagram-private-api');
const {BigQuery} = require('@google-cloud/bigquery');
async function exportData() {
// Replace these values with your own
const username = 'your_username';
const password = 'your_password';
const projectId = 'your_project_id';
const datasetId = 'your_dataset_id';
const tableId = 'your_table_id';
// Authenticate to Instagram and retrieve data
const device = new InstagramPrivateAPI.Device(username);
const storage = new InstagramPrivateAPI.CookieFileStorage(`${__dirname}/cookies/${username}.json`);
const session = await InstagramPrivateAPI.Session.create(device, storage, username, password);
// Use the Instagram API to retrieve data
const feed = new InstagramPrivateAPI.Feed.AccountFollowers(session);
const data = [];
let page = feed.iterate();
while (true) {
const {value} = await page.next();
if (!value) {
break;
}
data.push(value);
}
// Initialize the BigQuery client
const bigquery = new BigQuery({
projectId: projectId
});
// Load the data into a BigQuery table
const options = {
schema: 'name:string,username:string,profile_picture:string',
createDisposition: 'CREATE_IF_NEEDED',
writeDisposition: 'WRITE_APPEND',
};
const [job] = await bigquery
.dataset(datasetId)
.table(tableId)
.load(data, options);
console.log(`Job ${job.id} completed.`);
}
exportData();
Your code authenticates to Instagram using the instagram-private-api package and retrieves data from the user’s followers. It then uses the @google-cloud/bigquery package to create a new table in a BigQuery dataset and load the data into the table.
Keep in mind that you’ll need to replace the placeholder values in the code with your own Instagram username, password, and BigQuery project, dataset, and table IDs. You’ll also need to ensure that you have the necessary packages installed and that you have set up authorization for the BigQuery API.
To transfer data from Facebook to Google BigQuery, you can use the Facebook Graph API to obtain the data and then utilize the Google Cloud API to load it into BigQuery. This is a general overview of the steps involved in this process:
Create a Facebook developer account and obtain an access token that allows you to access the Facebook Graph API.
Use the Facebook Graph API to retrieve the data you want to export. You can use the API’s /{object-id}/{connection-name} endpoint to retrieve data for a specific object, such as a user or a page, and its connections, such as posts or comments.
Use the Google Cloud API to load the data into BigQuery. You can use the bq command-line tool or the BigQuery API to create a new table in BigQuery and load the data into it.
Here’s some example code using the request and google-auth-library libraries in Node.js to retrieve data from the Facebook Graph API and load it into BigQuery:
You’ll need to modify it to fit your specific use case.
For example, you may need to paginate through the results if you have more data than the API’s limit, and you’ll need to specify the correct object and connection names and fields for the data you want to retrieve.
You can find more information about the Facebook Graph API and the BigQuery API in the documentation linked below.
References:
Facebook Graph API documentation: https://developers.facebook.com/docs/graph-api
When building a data source on Google Sheets in Tableau Desktop 10mb is the max per Google Sheet. However, what if we could connect to more than one Google Sheet at the same time?
Google Sheets works wonders with Tableau Public because it allows for Tableau Public to read data from Google Sheets once per day. This enables everyone the capability to use an online cloud data source to update their Tableau Desktop dashboards.
Introduction
In this blog, we will discuss connecting to multiple Google Sheets in one connection. If “large sheets removed” is not sufficient, and you’re willing to break apart your sheets into multiple sheets manually or with an engineer, we will find this article helpful. We break apart each element, how it works, and explain how your engineer may begin breaking down the solution.
Tableau currently has no built in feature to allow this to happen, however they do have a feature you can setup to make it automatically connect to Google Sheets! Tableau suggests this isn’t possible and the only way to make it work is to use LESS DATA, but what if you have big data?
We built this blog to help you begin the journey of connecting to many sheets. You will want to demo this as a possible solution to show your engineering team to automatically create these Google Sheets (we are here to help too).
Error explained
If you begin connecting to a Google Sheet in Tableau Desktop >10mb, you will see various popups, depending on your operating system, explaining an error has occurred with Google Sheets.
Unable to complete action error message
Did you recently see this error?
Unable to complete action The Google Sheets service reported an unrecognized error when processing this request. This file is too large to be exported.
An error occurred while communicating with Google Sheets, <10mb Google sheets error message in Tableau desktop.
A good question to start asking, “will my data split into other sheets easily?”
In the example below we are going to speak towards an 80mb Google Sheet that will not work in Tableau.
TechWorkaround explained
The Tableau Desktop Wildcard (automatic) feature will capture Google Workbooks (contains google sheets) and google sheets in a workbook(s). It will “automate” building connections to multiple 10mb workbooks or sheets, by establishing a stack of data that will resemble your end goals. Similar to using Union All in SQL.
Stone Working Analogy
If you’re unfamiliar with union in SQL, think of your data stacking on each other like bricks on a brick wall. Data engineers and brick wall experts have similar jobs. They are solving a problem by stacking side by side or on top of each other.
We will focus on a “workbooks” example, where we will stack bricks of data on each other.
Using a matching pattern(xxx*), we are able to union similar named data storage.
Example; there are four regions in our data, each region is <10mbs, about 80mb total.
Four regions:
SOUTH 10mb
NORTH 10mb
EAST 10 mb
WEST 50 mb*
A Use Case Explaining the Solution
Step 0; union means the sheets need the same column headers. Thanks for learning about unions!
Step 1; build 8 googlesheets… (new workbooks, not new sheets, this works with sheets however I’m using workbooks for now)
Step2; name each google sheet workbook “Table-Demo_EXAMPLE” etc… and you will have the following.
Table-Demo_SOUTH
Table-Demo_NORTH
Table-Demo_EAST
Table-Demo_WEST_1
Table-Demo_WEST_2
Table-Demo_WEST_3
Table-Demo_WEST_4
Table123-Demo_WEST_5
Protip; Table123-Demo_WEST_5 will not be included in this exercise because it’s not named Table-Demo_. Wildcard allows you the ability to filter to the things you need. If you name your Google Sheets “Table-Demo_” our wildcard solution automates connection to that google sheet, there’s no need to connect to the extra google sheet if you’re setting up the solution as explained.
Now that we have an understanding of how a wildcard will work, let’s discuss the end to end.
How to setup >10mb union
To increase size of google sheets greater than 10 megabytes, and increase your overall Google Sheets insights in tableau desktop, you need to get good with the Union wildcard!
Connect to the googlesheet. Tableau desktop made this workflow a one click button on left side of opening screen. Requiring two clicks in total.
Walk through the Google authentication, choose which email with many similar tables for wildcard. This means you need to go and change the names of the Tables you wish to put together.
The renaming part needs to be a part of an automated process, you may want to do, using the Google Sheets API also known as the Google API, we found success automatically creating a new Google Sheet, and automatically naming the sheet similarly, which improved a client engagement during a tableau consulting engagement that had a lot of data engineering consulting to generate the solution. If data is constantly morphing, there may be a need to delete old sheets, we found clearing the sheet and re-populating data was the easiest method for fresh cleans. However lets get focused on the manual process because it’s a similar architecture. We found naming tables differently between tests helped us with testing/troubleshooting, and found Google Sheets had some strange hiccups that are easier to avoid by removing old tests completely.
Discussing Data Structure tips for >10mb Google Sheets
Here’s a good time to start making sure column headers are the same. If not it will continue to make a column, which will lead you down the path of learning how to solve for dynamic parameters due to string values being many to many.
Convert to union…
Very important step, drop down carrot and find the Convert to union click.
This workaround allows you to connect once, to all sheets similarly named (using wild cards) VS connecting to all the different google sheet workbooks. This allows you to remove many data sources and transition into one data source.
The Wildcard Union Screen in Tableau
Tableau offers a feature to union more than one google sheet together, which enables users to quickly build large cloud data storages on Tableau public, or internally.
Example; Tables-Demo_* will find anything with Tables-Demo_ as the start of the sheet name.
Helpful related resources and documentation.
Below are documents, notes, and community posts from other Tableau authors.
The 10mb limit with Google Sheets is ambiguous when testing the number with true CSV file sizes and better to determine a way of “stopping” the data before it gets big.
Some interesting things to think through, we found 7mb, 10.3mb, 12.9mb, and 19.1mb CSV files coming from single Google Sheet connections and no popup error stopping Tableau Desktop from connecting to the data. Don’t consider this size to be your break/test.
Screenshot demonstrating various CSV files downloaded from Google Sheets – Tested Oct 28, 2021
Good to note; This is the size of the csv when downloaded via the Google Sheets/Data/. Your team may get hung up on this process, and we found it’s better to focus on a row count if you’re not using full stack engineering to complete the task.
Thank you for reading. If you are interesting in implementing a process that uses Google API, contact us to learn more.
Researched & Written by, Tyler Garrett founder of Dev3lop | Consulting Services.
If you need to embed Google Data Studio reports in an iFrame on your website, we have that solution here with screenshots below.
How to embed google data studio steps.
Click File
Click Embed report
Click ‘copy to keyboard.’
That’s it. Celebrate. You’re done. It’s free and there’s no monthly fees or subscription.
Screenshots below!
The embed google data studio code – iframe
The code for embedding google data studio in an iframe is very easy.
<center><iframe style=”border: 0;” src=”http://XXXXXXX” width=”500″ height=”900″ frameborder=”0″ allowfullscreen=”allowfullscreen”></iframe></center>
Google Datas Studio Embed is packed with mobility.
Everyone is focused on reoccurring revenue and missed the functionality most users request. From easy embeds, to free ability to share across the organizations without a hefty pricing punch.
Google Data Studio has a lot of sharp swords in its disposal, for now lets focus on the mobility.
Squeeze down the browser to see how responsive everything is without any programming or clicks.
Google helps the world take a huge step in the right direction, as currently you have to build multiple iterations in Tableau Desktop – or your end users are stuck with a static mold or automatic sizing that doesn’t work for all devices because most users utilize large font sizes on their computer without even understanding what DPI settings are.
Hey, we only built this to offer a free solution because companies are trying to earn revenue off of this easy to do feature. Let us empower you to do this and you can save your bucks for another day.
Feel free to poke around after you finish embedding your google data studio report.
Embedding Google Data Studio Screenshots
Let us know if you need help. Advice is free! For solutions please see our business intelligence page!
Embed google data studio clickthrough. It’s only two clicks away!
Step 2 copy to clipboard! Embedding google data studio is too easy, thanks Google.
