Create single file in AWS Glue (pySpark) and store as custom file name S3

AWS Glue – AWS Glue is a serverless ETL tool developed by AWS. It is built on top of Spark. As spark is distributed processing engine by default it creates multiple output files states with e.g. Generating a Single file You might have requirement to create single output file. In order for you to create… Read More Create single file in AWS Glue (pySpark) and store as custom file name S3

Serverless – The provided execution role does not have permissions to call CreateNetworkInterface on EC2

Issue – while deploying Serverless Lambda (sls deploy command ) function I got below error message An error occurred: EsLambdaFunction – The provided execution role does not have permissions to call CreateNetworkInterface on EC2 (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: xxx). Resolution – In order to resolve the issue; I have… Read More Serverless – The provided execution role does not have permissions to call CreateNetworkInterface on EC2

AWS Lake formation – Error Message

Error Message – User: arn:aws:iam::xxxx:user/admin is not authorized to perform: lakeformation:PutDataLakeSettings with an explicit deny Issue – This issue will occur when you will try add new user as Lake formation admin with IAM user which already added as admin to Lake formation lakeformation:PutDataLakeSettings Denying this permission prevents a data lake administrator from designating other… Read More AWS Lake formation – Error Message

How to enable cron schedule in AWS EC2 instance

Step 1 – Connect to your ec2 instance ( follow below article ) https://learnsqlteam.com/2018/11/30/how-to-connect-aws-ec2-instance-through-terminal/ step 2 – go to /etc – ( cd /etc ) step 3 – edit crontab ( nano crontab ) step 4 – define schedule and command e.g.  in this example I am sysning file b/w ec2 instance and s3 bucket… Read More How to enable cron schedule in AWS EC2 instance