将Google数据存储备份从数据存储加载到Google BigQuery
问题描述:
我们的要求是以编程方式备份Google数据存储并将这些备份加载到Google Big查询以供进一步分析。我们成功地自动备份使用以下方法将Google数据存储备份从数据存储加载到Google BigQuery
Queue queue = QueueFactory.getQueue("datastoreBackupQueue");
/*
* Create a task which is equivalent to the backup URL mentioned in
* above cron.xml, using new queue which has Datastore admin enabled
*/
TaskOptions taskOptions = TaskOptions.Builder.withUrl("/_ah/datastore_admin/backup.create")
.method(TaskOptions.Method.GET).param("name", "").param("filesystem", "gs")
.param("gs_bucket_name",
"db-backup" + "/" + TimeUtils.parseDateToString(new Date(), "yyyy/MMM/dd"))
.param("queue", queue.getQueueName());
/*
* Get list of dynamic entity kind names from the datastore based on
* the kinds present in the datastore at the start of backup
*/
List<String> entityNames = getEntityNamesForBackup();
for (String entityName : entityNames) {
taskOptions.param("kind", entityName);
}
/* Add this task to above queue */
queue.add(taskOptions);
我能够再导入这个备份手动谷歌的BigQuery,但我们如何自动完成这一过程?
我也看过大多数的文档,并没有任何帮助 https://cloud.google.com/bigquery/docs/loading-data-cloud-storage#loading_data_from_google_cloud_storage
答
我已经解决了这个问题,下面是使用JAVA的解决方案 以下代码将从GoogleCloud存储中获取备份文件并将其加载到Google Big Query中。
AppIdentityCredential bqCredential = new AppIdentityCredential(
Collections.singleton(BigqueryScopes.BIGQUERY));
AppIdentityCredential dsCredential = new AppIdentityCredential(
Collections.singleton(StorageScopes.CLOUD_PLATFORM));
Storage storage = new Storage(HTTP_TRANSPORT, JSON_FACTORY, dsCredential);
Objects list = storage.objects().list(bucket).setPrefix(prefix).setFields("items/name").execute();
if (list == null) {
Log.severe(BackupDBController.class, "BackupToBigQueryController",
"List from Google Cloud Storage was null", null);
} else if (list.isEmpty()) {
Log.severe(BackupDBController.class, "BackupToBigQueryController",
"List from Google Cloud Storage was empty", null);
} else {
for (String kind : getEntityNamesForBackup()) {
Job job = new Job();
JobConfiguration config = new JobConfiguration();
JobConfigurationLoad loadConfig = new JobConfigurationLoad();
String url = "";
for (StorageObject obj : list.getItems()) {
String currentUrl = obj.getName();
if (currentUrl.contains(kind + ".backup_info")) {
url = currentUrl;
break;
}
}
if (StringUtils.isStringEmpty(url)) {
continue;
} else {
url = "gs://"+bucket+"/" + url;
}
List<String> gsUrls = new ArrayList<>();
gsUrls.add(url);
loadConfig.setSourceUris(gsUrls);
loadConfig.set("sourceFormat", "DATASTORE_BACKUP");
loadConfig.set("allowQuotedNewlines", true);
TableReference table = new TableReference();
table.setProjectId(projectId);
table.setDatasetId(datasetId);
table.setTableId(kind);
loadConfig.setDestinationTable(table);
config.setLoad(loadConfig);
job.setConfiguration(config);
Bigquery bigquery = new Bigquery.Builder(HTTP_TRANSPORT, JSON_FACTORY, bqCredential)
.setApplicationName("BigQuery-Service-Accounts/0.1").setHttpRequestInitializer(bqCredential)
.build();
Insert insert = bigquery.jobs().insert(projectId, job);
JobReference jr = insert.execute().getJobReference();
Log.info(BackupDBController.class, "BackupToBigQueryController",
"Moving data to BigQuery was successful", null);
}
}
如果任何人有一个更好的方法,请让我知道
答
在,你在你的问题中提到的loading data from Google Cloud Storage article,仅描述GCS进口的一些纲领性的例子是使用命令行,Node.js的或蟒蛇。
您还可以自动位于云存储BigQuery的进口数据,通过在脚本中运行以下命令:
$ gcloud alpha bigquery import SOURCE DESTINATION_TABLE
有关该命令访问此article更多信息。
答
截至上周有自动执行此之有道。最重要的部分是gcloud beta datastore export
。
我创造了周围的短脚本:https://github.com/chees/datastore2bigquery
您可以调整,以适应您的情况。