How do I troubleshoot errors when I run SageMaker training jobs?

Key Takeaways

[Music] [Applause] [Music] Hello, I'm Juan, a cloud support engineer here at the AWS office in Oakuckland. Today I'm going to show you how to troubleshoot errors when running Amazon SageMaker training jobs. Let's get started. If your SageMaker training job fails, then the first step is to check the failure reason on the SageMaker console or you can use the describe training job API call. First, navigate to SageMaker AI, choose training and training jobs. If the job failed with an error, then first retry the job to rule out any transient issues. To do this, select clone and then choose create training job. If the job fails again after retrying, then we must review the logs in Amazon Cloudatch. Open the Cloudatch console, click log groups, and locate the log group/ AWS/SageMaker/training jobs. Here you will find the log streams for each training job, including the job name, instance number, and timestamp. Review the logs for any error messages or failures. You can also check job metrics such as CPU utilization, memory utilization, disk utilization, GPU utilization, and GPU memory utilization to see if the failure was because of resource limitations. Let's go back to the SageMaker console and select a failed training job. In the monitor section, you can view the instance utilization graphs. If the metric shows that job used up all available resources, then consider switching to a larger instance type or attaching a larger storage volume to the instance for your next training job. In summary, check the failure reason, review Cloudatch logs and metrics, and adjust computer resources if needed to resolve transient issues such as throttling errors or temporary capacity constraints. You can retry the job. And now you know how to troubleshoot errors when running Amazon SageMaker training jobs. Thanks for watching and happy cloud computing from all of us here at AWS. [Applause] [Music]