관찰과 질문, 그리고 데이터

spark 4.x.x 버전에서 사내 s3-compatible object storage로 history server를 연결할 때 생겼던 문제 트러블슈팅 기록.문제spark 3.5 버전과 spark 4.1 버전을 사용하는데, 동일한 spark-default.conf 를 사용하면 spark 4.1 이미지에서만 아래와 같은 에러가 난다.# spakr 3.5.7 이미지FROM apache/spark:3.5.7USER root# Hadoop 3.3.4 dependenciesENV HADOOP_AWS_VERSION=3.3.4# AWS SDK V1 bundle compatible with Hadoop 3.3.4ENV AWS_SDK_V1_VERSION=1.12.262# Dataflint version for enhan..

https://youtu.be/7yVXMCX62tY?si=N-AivSULV2dG6tjdAdobe 사내의 GitOps CI/CD 툴 이름이 Flex.Source인 Git에 저장된 정보를 Target인 여러 destination에 배포하기 위한 서비스.ArgoCD, Argo Workflows, Argo Events, Custom Components... 등 다양한 프로덕트가 통합되어 있음.Template 제공, Fully Customizable CI pipelineJust-in-Time Provisioning.Advanced Deployment Strategy (Argo Rollout같은 거)"Single pane of glass": 모든 프로젝트 정보를 볼 수 있는 service Management too..

Study Materials Coursera: https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org udemy: https://www.udemy.com/course/practice-exams-aws-certified-solutions-architec..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org Processing Data in the Data Lake Data Prep and AWS Glue Jobs Data Ingestion in raw format. 사용하고자 하는 목적에 맞게 da..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org Ingesting the River Use the right tool for the job Data categorizing: 필요할 때 원하는 데이터를 정확히 찾아내기 위해 필요함. 사용자의 ac..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org AWS Data Related Services 다양한 AWS Service를 일종의 모듈 개념으로 조합해서 Data Lake를 구성할 수 있다. 핵심 컴포넌트 위주로 먼저 보고, 해당 컴포넌트의 ..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org Introduction to Data lakes Data Lake가 필요한 대표적인 시나리오는 아래와 같다. 예컨대 startup의 System Administrator라고 가정해보자. 목표는 E..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org Customer #4 - requirements 모든 서비스가 하나의 AWS Account (prod account) 내에 구성돼 있음. divide big single account into s..

https://www.coursera.org/professional-certificates/aws-cloud-solutions-architect AWS Cloud Solutions Architect Amazon Web Services에서 제공합니다. Start here to become an AWS Solutions Architect. Gain the skills and knowledge to design architectural solutions on ... 무료로 등록하십시오. www.coursera.org Designing a hybrid solution for container based workloads on AWS Customer #3 - requirements 전체 workload의 절반은 ..

AWS X-Ray & Amazon CloudWatch: Increasing visibility for ECS & Fargatehttps://youtu.be/qXLsx4r37_0 Vanguard: 전세계 170여개 국, 3천만 investors에게 Mutual Fund / ETF 판매하고 Financial advice 제공하는 회사.2019년 경 AWS ECS on Fargate로 MicroService 도입. underlying infra 고민 없이 high performace application / business logic 구현에 신경쓸 수 있도록Vanguard에서 legacy로 사용하던 monitoring / logging tool의 Leverage...cloud-native Observabili..