spark-observability-skills

Name: spark-observability-skills
Author: EmbrasureAI

Pending

Open-source agent skills for debugging and optimizing Apache Spark workloads

50stars

13forks

Python

Installation

# Add to your Claude Code skills
git clone https://github.com/EmbrasureAI/spark-observability-skills

Getting Started

Guides for using ai agents skills like spark-observability-skills.

Caveman: Cut Claude Token Use by 65%
How agent-side prompt compression works, when to use it, and when not to.
What is an AI Skills Marketplace?
Definitions, how marketplaces work, and how to choose between them in 2026.
Getting Started with AI Skills

README.md

Frequently Asked Questions

What is spark-observability-skills?

spark-observability-skills is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by EmbrasureAI. Open-source agent skills for debugging and optimizing Apache Spark workloads. It has 50 GitHub stars.

Is spark-observability-skills safe to use?

spark-observability-skills's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.

How do I install spark-observability-skills?

Clone the repository with "git clone https://github.com/EmbrasureAI/spark-observability-skills" and add it to your Claude Code skills directory (see the Installation section above).

What programming language is spark-observability-skills written in?

spark-observability-skills is primarily written in Python. It is open-source under EmbrasureAI on GitHub, so you can review or fork the full source.

Are there alternatives to spark-observability-skills?

Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh spark-observability-skills against similar tools.

Agentic AI for Beginners

Build your first AI agent from scratch - tool use, ReAct pattern, memory, deployment

41 minBeginner

Comments (0)

to leave a comment.

No comments yet. Be the first to share your thoughts!

Related Skills

ECC

by affaan-m

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

243,639

soul-grader-skill swarm-factory

⚠️ Third-Party Software Notice

This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.

Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.

Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.

Read the Terms of Service

Spark observability skills

Open-source agent skills from Embrasure for diagnosing and optimizing Apache Spark workloads. Each skill is a single SKILL.md with an ordered list of the highest-impact causes to check, plus a read-only Spark History Server REST client under scripts/ that collects the runtime evidence in one bounded snapshot.

Install

git clone https://github.com/EmbrasureAI/spark-observability-skills.git
ln -s "$PWD"/spark-observability-skills/skills/* ~/.codex/skills/

Point the symlinks at whichever skills directory your harness reads (~/.codex/skills, ~/.claude/skills, ...), creating it first if needed, then restart or reload the harness.

Or paste this into your agent:

Clone https://github.com/EmbrasureAI/spark-observability-skills and symlink each directory under skills/ into your skills directory, then tell me to reload.

Setup

The skills need HTTP access to a Spark History Server, or to the live UI of a running application (the driver UI on port 4040 serves the same REST API):

export SPARK_HISTORY_URL="https://<your-history-server>"   # e.g. http://localhost:18080 locally or via a tunnel
export SPARK_HISTORY_AUTHORIZATION="Bearer <token>"        # only if the server requires auth

History data exists only for applications that ran with spark.eventLog.enabled=true.
If the server is cluster-internal, open a tunnel first, for example kubectl port-forward svc/spark-history-server 18080:18080.
Behind an SSO proxy, reuse your browser session with SPARK_HISTORY_COOKIE or SPARK_HISTORY_HEADERS_JSON; pass --ca-file for a private CA.

Skills

Debug Spark failures: a run failed. Trace driver and executor crashes, out-of-memory kills, fetch failures, task exceptions, and aborted stages back to the earliest supported cause instead of the last retry error.
Debug slow Spark jobs: a run is slower or more expensive than it should be. Compare against a healthy run to localize the first divergence: skew, shuffle, spill, GC, poor parallelism, scheduler delay, or infrastructure.
Optimize Spark SQL plans: a query works but costs too much. Read its final adaptive plan and runtime metrics to cut scans, shuffles, joins, and unnecessary work without changing query results.

Safety

The collector is read-only, bounds large responses by default, redacts sensitive Spark properties, and keeps TLS verification enabled. Review every command against your environment and access policies before running it.

Contributing

Each skill directory is self-contained so it can be symlinked or copied on its own. As a result, scripts/spark_history_api.py is intentionally identical across the three skills. If you change one copy, sync all three.

License

Apache-2.0. Apache Spark, Apache Celeborn, and their respective marks belong to the Apache Software Foundation. This project is not an official Apache Software Foundation project.