Minio

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.

Objetivo

Documentar instalação e armazenamento de arquivos parquet.

Links

[1] Docker compose: https://github.com/starburstdata/dbt-trino
[2] Trino TPCH: https://trino.io/docs/current/connector/tpch.html#sql-support
[3] Create Schema: https://github.com/sensei23/trino-hive-docker/tree/main

Dependências

[1]

Instalação (Docker)

Com o arquivo de docker compose será criado 4 containters: trino, postgres, hive e minio.

# Subir containers
docker compose up -d

Trino para consultas

Acesse o serviço do trino dentro do container e realize consultas:

# Mostra catalogs encontrados pelo trino
SHOW CATALOGS;

# Cria esquema
CREATE SCHEMA iceberg.tpch
WITH (location = 's3a://datalake/');

# Cria tabela com dados no minio e metadata no hive
CREATE TABLE iceberg.tpch.lineitem
WITH (
    format = 'PARQUET',
    location = 's3a://datalake/lineitem/'
)
AS SELECT * FROM tpch.sf10.lineitem;

# Consulta dados
SELECT DISTINCT returnflag FROM iceberg.tpch.lineitem;