Release from Development#67
Conversation
…o fix HTML minifier warnings
…dd upstream sync step
Feat/UI design
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (19)
WalkthroughThis PR rewrites multiple documentation pages for DeepScanBot's Docusaurus site: Code of Conduct, contribution guide, installation guide, features guide, usage guide, and introduction page. Content is reorganized into new sections with updated examples, card-style JSX layouts, revised CLI flag tables, and expanded workflow instructions. No code entities changed. ChangesDocumentation restructuring
Estimated code review effort: 2 (Simple) | ~15 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/docs/guide/features.mdx`:
- Around line 119-143: The CLI Reference table’s `-host-concurrency` entry is
missing the fallback clarification used elsewhere, so update that row in the CLI
Reference section to state that `0` uses `-concurrency`. Keep the wording
aligned with the existing `-concurrency` description and the flag help text so
the behavior is clear and consistent.
In `@apps/docs/docs/introduction.mdx`:
- Around line 19-152: The docs page repeats the same card wrapper styles across
the feature and next-step sections, so extract that shared layout into a
reusable component or CSS class. Update the introduction page to use a single
`FeatureCard`/`NextStepCard`-style component (or shared class) for the repeated
`div` structure, keeping the existing `title`, `icon`, `description`, and
optional `href` content as props. This will centralize the padding, border,
radius, and spacing logic and avoid maintaining the same inline styles in
multiple JSX blocks.
- Around line 81-82: The sentence in the docs contains a stray mid-sentence
capitalization in the phrase from the introduction copy. Update the text in the
introduction content so “Smart Retry-After header parsing” reads as a normal
sentence fragment with consistent capitalization, matching the surrounding prose
in the same section.
- Around line 192-256: Update the docs navigation links in the introduction page
to use Docusaurus routing instead of hardcoded /docs/... anchors, since those
links bypass the site base path and can break in deployment. In the card
sections for Installation, Usage Guide, and Features, replace the current
anchor-based navigation with Docusaurus Link-based doc navigation (or equivalent
relative doc links) so the baseUrl is applied automatically; use the existing
link/card markup in introduction.mdx as the place to make the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6bae3b79-075d-4e62-9c1b-89f5b943e629
📒 Files selected for processing (6)
apps/docs/docs/contribution-guide/code-of-conduct.mdxapps/docs/docs/contribution-guide/how-to-contribute.mdxapps/docs/docs/guide/features.mdxapps/docs/docs/guide/usage.mdxapps/docs/docs/installation.mdxapps/docs/docs/introduction.mdx
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Build & Test
⚠️ CI failures not shown inline (2)
GitHub Actions: CI / Build & Test: feat: Merge pull request #66 from mindfiredigital/feat/ui-design
Conclusion: failure
##[group]Run go test -v -count=1 ./...
�[36;1mgo test -v -count=1 ./...�[0m
�[36;1mecho "✅ All tests passed"�[0m
shell: /usr/bin/bash -e {0}
##[endgroup]
? github.com/mindfiredigital/DeepScanBot/apps/cli [no test files]
=== RUN TestCrawlerStartReturnsResultsWithoutWritingFiles
level=INFO msg="Starting crawl: url=http://127.0.0.1:40707 max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:40707 (depth=0)"
level=INFO msg="Crawled http://127.0.0.1:40707 [status=200] [result=passed]"
level=INFO msg="Crawl finished: url=http://127.0.0.1:40707 total=1 passed=1 failed=0 skipped=0 duration=1ms"
--- PASS: TestCrawlerStartReturnsResultsWithoutWritingFiles (0.00s)
=== RUN TestCrawlerRespectsRobots
level=INFO msg="Starting crawl: url=http://127.0.0.1:36855/public max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:36855/public (depth=0)"
level=INFO msg="Crawled http://127.0.0.1:36855/public [status=200] [result=passed]"
level=INFO msg="Crawl finished: url=http://127.0.0.1:36855/public total=1 passed=1 failed=0 skipped=0 duration=1ms"
level=INFO msg="Starting crawl: url=http://127.0.0.1:36855/private max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:36855/private (depth=0)"
level=INFO msg="Skipping http://127.0.0.1:36855/private because robots.txt disallows it"
level=INFO msg="Crawl finished: url=http://127.0.0.1:36855/private total=1 passed=0 failed=0 skipped=1 duration=0s"
--- PASS: TestCrawlerRespectsRobots (0.00s)
=== RUN TestCrawlerProcessesDiscoveredLinks
level=INFO msg="Starting crawl: url=http://127.0.0.1:45755 max-depth=1 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:45755 (depth=0)"
level=INFO msg="Crawled http:/...
GitHub Actions: CI / 0_Build & Test.txt: feat: Merge pull request #66 from mindfiredigital/feat/ui-design
Conclusion: failure
Current runner version: '2.335.1'
##[group]Runner Image Provisioner
Hosted Compute Agent
Version: 20260624.560
Commit: 925d229a51159bc391ae97e54a2dd1fe20af789d
Build Date:
Worker ID: {5bd8a0fc-ff05-4a75-98e6-0c36ea7fdfd9}
Azure Region: northcentralus
##[endgroup]
##[group]Operating System
Ubuntu
24.04.4
LTS
##[endgroup]
##[group]Runner Image
Image: ubuntu-24.04
Version: 20260628.225.1
Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20260628.225/images/ubuntu/Ubuntu2404-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20260628.225
##[endgroup]
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
##[endgroup]
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5)
Download action repository 'actions/setup-go@v5' (SHA:40f1582b2485089dde7abd97c1529aa768e1baff)
Complete job name: Build & Test
Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
##[group]Run actions/checkout@v4
with:
fetch-depth: 0
repository: mindfiredigital/DeepScanBot
***REDACTED***
ssh-strict: true
ssh-user: git
persist-credentials: true
clean: true
sparse-checkout-cone-mode: true
fetch-tags: false
show-progress: true
lfs: false
submodules: false
set-safe-directory: true
##[endgroup]
Syncing repository: mindfiredigital/DeepScanBot
##[group]Getting Git version info
Working directory is '/home/runner/work/DeepScanBot/DeepScanBot'
[command]/usr/bin/git version
git version 2.54.0
##[endgroup]
Temporarily overriding HOME='/home/runner/work/_t...
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
**: # Contributing to DeepScanBotThank you for your interest in contributing to DeepScanBot! This document provides guidelines and instructions for contributing to this project.
Table of Contents
- Code of Conduct
- Getting Started
- Development Setup
- How to Contribute
- Pull Request Process
- Coding Standards
- Testing
- Reporting Bugs
- Suggesting Features
Code of Conduct
By participating in this project, you agree to abide by the CODE_OF_CONDUCT.md. Please be respectful and constructive in all interactions.
Getting Started
- Fork the repository on GitHub
- Clone your fork locally:
git clone https://github.com/YOUR_USERNAME/DeepScanBot.git cd DeepScanBot- Add the upstream remote:
git remote add upstream https://github.com/mindfiredigital/DeepScanBot.gitDevelopment Setup
Prerequisites
- Go (version 1.21 or higher)
- Git
Installation
Install dependencies:
go mod downloadInstall development tools:
go install mvdan.cc/gofumpt@latest go install github.com/daixiang0/gci@latest go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest go install github.com/evilmartians/lefthook@latestInstall git hooks:
lefthook installBuild the project:
go build -o deepscanbot .Verify the installation:
go run ./apps/cli -hProject Structure
DeepScanBot/ ├── crawler/ # Core crawling logic and orchestration ├── fetcher/ # HTTP fetching and content-type filtering ├── parser/ # HTML parsing and link extraction ├── storage/ # Data storage and report generation ├── logger/ # Logging utilitie...
Files:
apps/docs/docs/contribution-guide/code-of-conduct.mdxapps/docs/docs/installation.mdxapps/docs/docs/contribution-guide/how-to-contribute.mdxapps/docs/docs/guide/features.mdxapps/docs/docs/introduction.mdxapps/docs/docs/guide/usage.mdx
🪛 LanguageTool
apps/docs/docs/contribution-guide/code-of-conduct.mdx
[style] ~26-~26: Try using a synonym here to strengthen your wording.
Context: ...ces - Trolling, insulting or derogatory comments, and personal or political attacks - Pu...
(COMMENT_REMARK)
🔇 Additional comments (12)
apps/docs/docs/contribution-guide/code-of-conduct.mdx (1)
7-45: LGTM!apps/docs/docs/contribution-guide/how-to-contribute.mdx (3)
7-59: LGTM!
108-114: LGTM!
71-77: 📐 Maintainability & Code QualityNo change needed for the
gciexample. The module path and lefthook import command already useprefix(github.com/mindfiredigital/DeepScanBot), so this doc matches the repo’s import-grouping setup.> Likely an incorrect or invalid review comment.apps/docs/docs/installation.mdx (5)
87-89: 📐 Maintainability & Code Quality | ⚡ Quick win
<p>tags reintroduced in JSX card blocks.The PR's own commits fix HTML-minifier warnings elsewhere by replacing
<p>with<div>in JSX/MDX. These new card blocks still use<p>inside<div>containers (single-line, so likely safe from the nested-paragraph MDX bug, but inconsistent with the stated project convention going forward).♻️ Align with the `
` convention used elsewhere in this PR- <a href="/docs/guide/usage"><strong>Usage Guide</strong></a> - <p style={{ fontSize: '0.875rem', margin: '0.25rem 0 0', color: 'var(--ifm-color-emphasis-600)' }}>Learn how to use all CLI flags</p> + <a href="/docs/guide/usage"><strong>Usage Guide</strong></a> + <div style={{ fontSize: '0.875rem', margin: '0.25rem 0 0', color: 'var(--ifm-color-emphasis-600)' }}>Learn how to use all CLI flags</div>- <a href="/docs/guide/features"><strong>Features</strong></a> - <p style={{ fontSize: '0.875rem', margin: '0.25rem 0 0', color: 'var(--ifm-color-emphasis-600)' }}>Explore all features in depth</p> + <a href="/docs/guide/features"><strong>Features</strong></a> + <div style={{ fontSize: '0.875rem', margin: '0.25rem 0 0', color: 'var(--ifm-color-emphasis-600)' }}>Explore all features in depth</div>Also applies to: 100-102
7-33: LGTM!
34-56: Solid verification flow.Build description,
go runalternative, and verify/smoke-test commands align well with the CLI contract (-url,-depth,-json,-h) and CI's build/test steps.
57-86: LGTM!
15-15: 🎯 Functional CorrectnessGo 1.22+ matches the module target.
> Likely an incorrect or invalid review comment.apps/docs/docs/introduction.mdx (1)
7-16: LGTM!Also applies to: 154-176
apps/docs/docs/guide/features.mdx (1)
2-118: LGTM!apps/docs/docs/guide/usage.mdx (1)
2-9: LGTM!Also applies to: 28-66, 80-80, 90-113, 191-191
| ## CLI Reference | ||
|
|
||
| | Flag | Default | Description | | ||
| |------|---------|-------------| | ||
| | `-url` | `""` | Starting URL | | ||
| | `-url` | `""` | Starting URL **(required)** | | ||
| | `-depth` | `2` | Maximum crawl depth | | ||
| | `-timeout` | `2` | Request timeout (seconds) | | ||
| | `-proxy` | `""` | Proxy URL | | ||
| | `-json` | `false` | JSON output | | ||
| | `-size` | `-1` | Page size limit (KB) | | ||
| | `-dr` | `false` | Disable redirects | | ||
| | `-s` | `false` | Show URL source | | ||
| | `-insecure` | `false` | Disable TLS verification | | ||
| | `-u` | `false` | Unique URLs only | | ||
| | `-concurrency` | `0` | Max concurrent requests | | ||
| | `-timeout` | `2` | Request timeout in seconds | | ||
| | `-concurrency` | `0` | Max concurrent workers (0 = CPU cores) | | ||
| | `-host-concurrency` | `0` | Max concurrent requests per host | | ||
| | `-content-types` | `"text/html"` | Allowed MIME types | | ||
| | `-output` | `"crawler_results"` | Output filename | | ||
| | `-ignore-robots` | `false` | Ignore robots.txt | | ||
| | `-cross-domain` | `false` | Follow external links | | ||
| | `-json` | `false` | JSON output | | ||
| | `-size` | `-1` | Page size limit in KB | | ||
| | `-proxy` | `""` | Proxy URL | | ||
| | `-u` | `false` | Unique URLs only | | ||
| | `-s` | `false` | Show URL source | | ||
| | `-dr` | `false` | Disable redirects | | ||
| | `-insecure` | `false` | Disable TLS verification | | ||
| | `-retries` | `0` | Retry attempts | | ||
| | `-retry-backoff` | `"1s"` | Retry backoff duration | | ||
| | `-delay` | `"0s"` | Politeness delay | | ||
| | `-sitemap` | `false` | Discover sitemap.xml | | ||
| | `-resume` | `false` | Resume mode | | ||
| | `-ignore-robots` | `false` | Ignore robots.txt | | ||
| | `-cross-domain` | `false` | Follow external links | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
-host-concurrency row drops the "0 = uses -concurrency" clarification.
The README's CLI flags table and the main.go flag help text both spell out that a value of 0 falls back to -concurrency: "Maximum concurrent requests per host; 0 uses -concurrency". The new CLI Reference table only says "Max concurrent requests per host" — readers may not realize 0 isn't "no limit" but rather "inherit from -concurrency". Worth restoring that clarification for parity with the -concurrency row, which does keep its "(0 = CPU cores)" note.
📝 Proposed fix
-| `-host-concurrency` | `0` | Max concurrent requests per host |
+| `-host-concurrency` | `0` | Max concurrent requests per host (0 = uses -concurrency) |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ## CLI Reference | |
| | Flag | Default | Description | | |
| |------|---------|-------------| | |
| | `-url` | `""` | Starting URL | | |
| | `-url` | `""` | Starting URL **(required)** | | |
| | `-depth` | `2` | Maximum crawl depth | | |
| | `-timeout` | `2` | Request timeout (seconds) | | |
| | `-proxy` | `""` | Proxy URL | | |
| | `-json` | `false` | JSON output | | |
| | `-size` | `-1` | Page size limit (KB) | | |
| | `-dr` | `false` | Disable redirects | | |
| | `-s` | `false` | Show URL source | | |
| | `-insecure` | `false` | Disable TLS verification | | |
| | `-u` | `false` | Unique URLs only | | |
| | `-concurrency` | `0` | Max concurrent requests | | |
| | `-timeout` | `2` | Request timeout in seconds | | |
| | `-concurrency` | `0` | Max concurrent workers (0 = CPU cores) | | |
| | `-host-concurrency` | `0` | Max concurrent requests per host | | |
| | `-content-types` | `"text/html"` | Allowed MIME types | | |
| | `-output` | `"crawler_results"` | Output filename | | |
| | `-ignore-robots` | `false` | Ignore robots.txt | | |
| | `-cross-domain` | `false` | Follow external links | | |
| | `-json` | `false` | JSON output | | |
| | `-size` | `-1` | Page size limit in KB | | |
| | `-proxy` | `""` | Proxy URL | | |
| | `-u` | `false` | Unique URLs only | | |
| | `-s` | `false` | Show URL source | | |
| | `-dr` | `false` | Disable redirects | | |
| | `-insecure` | `false` | Disable TLS verification | | |
| | `-retries` | `0` | Retry attempts | | |
| | `-retry-backoff` | `"1s"` | Retry backoff duration | | |
| | `-delay` | `"0s"` | Politeness delay | | |
| | `-sitemap` | `false` | Discover sitemap.xml | | |
| | `-resume` | `false` | Resume mode | | |
| | `-ignore-robots` | `false` | Ignore robots.txt | | |
| | `-cross-domain` | `false` | Follow external links | | |
| ## CLI Reference | |
| | Flag | Default | Description | | |
| |------|---------|-------------| | |
| | `-url` | `""` | Starting URL **(required)** | | |
| | `-depth` | `2` | Maximum crawl depth | | |
| | `-timeout` | `2` | Request timeout in seconds | | |
| | `-concurrency` | `0` | Max concurrent workers (0 = CPU cores) | | |
| | `-host-concurrency` | `0` | Max concurrent requests per host (0 = uses -concurrency) | | |
| | `-content-types` | `"text/html"` | Allowed MIME types | | |
| | `-output` | `"crawler_results"` | Output filename | | |
| | `-json` | `false` | JSON output | | |
| | `-size` | `-1` | Page size limit in KB | | |
| | `-proxy` | `""` | Proxy URL | | |
| | `-u` | `false` | Unique URLs only | | |
| | `-s` | `false` | Show URL source | | |
| | `-dr` | `false` | Disable redirects | | |
| | `-insecure` | `false` | Disable TLS verification | | |
| | `-retries` | `0` | Retry attempts | | |
| | `-retry-backoff` | `"1s"` | Retry backoff duration | | |
| | `-delay` | `"0s"` | Politeness delay | | |
| | `-sitemap` | `false` | Discover sitemap.xml | | |
| | `-resume` | `false` | Resume mode | | |
| | `-ignore-robots` | `false` | Ignore robots.txt | | |
| | `-cross-domain` | `false` | Follow external links | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/docs/guide/features.mdx` around lines 119 - 143, The CLI Reference
table’s `-host-concurrency` entry is missing the fallback clarification used
elsewhere, so update that row in the CLI Reference section to state that `0`
uses `-concurrency`. Keep the wording aligned with the existing `-concurrency`
description and the flag help text so the behavior is clear and consistent.
| <div className="row"> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>⚡ Concurrent Crawling</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Multi-threaded architecture with configurable worker pools, per-host | ||
| rate limiting, and CPU-aware auto-scaling. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>🛡️ Robots.txt Compliance</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Automatically respects robots.txt rules with an option to bypass when | ||
| needed. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>🔄 Retry & Rate-Limit Handling</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Automatic retry with exponential backoff, Smart Retry-After header | ||
| parsing, and configurable politeness delays. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>📊 Rich Output</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| JSON or text reports with detailed summaries, status code distribution, | ||
| skip reason breakdowns, and retry analytics. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>🗺️ Sitemap Discovery</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Auto-discover and crawl URLs from sitemap.xml, including nested sitemap | ||
| indexes. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--6" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1.25rem", | ||
| borderRadius: "0.75rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| height: "100%", | ||
| }} | ||
| > | ||
| <strong>📁 Content-Type Filtering</strong> | ||
| <div | ||
| style={{ | ||
| margin: "0.5rem 0 0", | ||
| fontSize: "0.9375rem", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Filter downloads by MIME type, enforce page size limits, and focus on | ||
| specific content types. | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Extract repeated card styling into a shared component.
Nine near-identical JSX blocks (six feature cards + three "next steps" cards) duplicate the same inline style={{...}} objects for padding, border, radius, etc. Classic DRY violation — any future style tweak (e.g., border color, radius) means updating 9 places and risking drift.
Consider a small reusable <FeatureCard> / <NextStepCard> React component (or a shared CSS class) that takes title, icon, description, and optional href props.
♻️ Example component sketch
// src/components/FeatureCard.jsx
export default function FeatureCard({ icon, title, children }) {
return (
<div className="col col--6" style={{ marginBottom: "1rem" }}>
<div
style={{
padding: "1.25rem",
borderRadius: "0.75rem",
border: "1px solid var(--ifm-color-emphasis-200)",
height: "100%",
}}
>
<strong>{icon} {title}</strong>
<div style={{ margin: "0.5rem 0 0", fontSize: "0.9375rem", color: "var(--ifm-color-emphasis-600)" }}>
{children}
</div>
</div>
</div>
);
}Also applies to: 180-256
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/docs/introduction.mdx` around lines 19 - 152, The docs page repeats
the same card wrapper styles across the feature and next-step sections, so
extract that shared layout into a reusable component or CSS class. Update the
introduction page to use a single `FeatureCard`/`NextStepCard`-style component
(or shared class) for the repeated `div` structure, keeping the existing
`title`, `icon`, `description`, and optional `href` content as props. This will
centralize the padding, border, radius, and spacing logic and avoid maintaining
the same inline styles in multiple JSX blocks.
| Automatic retry with exponential backoff, Smart Retry-After header | ||
| parsing, and configurable politeness delays. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Typo: stray capitalization mid-sentence.
"Smart Retry-After header parsing" — "Smart" is capitalized oddly mid-sentence, reads as a leftover heading/label fragment.
✏️ Suggested fix
- Automatic retry with exponential backoff, Smart Retry-After header
+ Automatic retry with exponential backoff, smart Retry-After header
parsing, and configurable politeness delays.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Automatic retry with exponential backoff, Smart Retry-After header | |
| parsing, and configurable politeness delays. | |
| Automatic retry with exponential backoff, smart Retry-After header | |
| parsing, and configurable politeness delays. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/docs/introduction.mdx` around lines 81 - 82, The sentence in the
docs contains a stray mid-sentence capitalization in the phrase from the
introduction copy. Update the text in the introduction content so “Smart
Retry-After header parsing” reads as a normal sentence fragment with consistent
capitalization, matching the surrounding prose in the same section.
| <a href="/docs/installation"> | ||
| <strong>Installation</strong> | ||
| </a> | ||
| <div | ||
| style={{ | ||
| fontSize: "0.875rem", | ||
| margin: "0.25rem 0 0", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Set up DeepScanBot on your machine | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--4" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1rem", | ||
| borderRadius: "0.5rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| textAlign: "center", | ||
| background: "var(--ifm-card-background-color)", | ||
| }} | ||
| > | ||
| <div style={{ fontSize: "1.5rem", marginBottom: "0.5rem" }}>📖</div> | ||
| <a href="/docs/guide/usage"> | ||
| <strong>Usage Guide</strong> | ||
| </a> | ||
| <div | ||
| style={{ | ||
| fontSize: "0.875rem", | ||
| margin: "0.25rem 0 0", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Learn how to use all CLI flags | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div className="col col--4" style={{ marginBottom: "1rem" }}> | ||
| <div | ||
| style={{ | ||
| padding: "1rem", | ||
| borderRadius: "0.5rem", | ||
| border: "1px solid var(--ifm-color-emphasis-200)", | ||
| textAlign: "center", | ||
| background: "var(--ifm-card-background-color)", | ||
| }} | ||
| > | ||
| <div style={{ fontSize: "1.5rem", marginBottom: "0.5rem" }}>✨</div> | ||
| <a href="/docs/guide/features"> | ||
| <strong>Features</strong> | ||
| </a> | ||
| <div | ||
| style={{ | ||
| fontSize: "0.875rem", | ||
| margin: "0.25rem 0 0", | ||
| color: "var(--ifm-color-emphasis-600)", | ||
| }} | ||
| > | ||
| Explore all features in depth | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check configured baseUrl for the docs site
fd docusaurus.config apps/docs -x cat {}Repository: mindfiredigital/DeepScanBot
Length of output: 1952
Switch these doc links to Docusaurus routing. The /docs/... anchors bypass baseUrl (/DeepScanBot/) and will 404 on the deployed site; use <Link to="..."> or relative doc links so Docusaurus prefixes the base path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/docs/introduction.mdx` around lines 192 - 256, Update the docs
navigation links in the introduction page to use Docusaurus routing instead of
hardcoded /docs/... anchors, since those links bypass the site base path and can
break in deployment. In the card sections for Installation, Usage Guide, and
Features, replace the current anchor-based navigation with Docusaurus Link-based
doc navigation (or equivalent relative doc links) so the baseUrl is applied
automatically; use the existing link/card markup in introduction.mdx as the
place to make the change.
…aser to convert binary to npm packages
- Remove all Go references from docs (users interact via npm only) - Update package.json with enhanced description, keywords, and release config - Update release.yml with Deployment Workflow pattern (semantic-release) - Enhance introduction, installation, features, and usage docs - Remove outdated docs/ directory files - Update homepage meta description
Feat/binary setup
There was a problem hiding this comment.
Actionable comments posted: 21
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/cli/tests/main_test.go (1)
13-77: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMissing tests for
version,doctor,config, andcompletionsubcommands.The scan command and help flag are well-tested, but the newly added subcommands have no coverage. Per coding guidelines, unit tests should be written for new functionality.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/cli/tests/main_test.go` around lines 13 - 77, The CLI test suite currently covers scan and help only; add unit coverage for the new `version`, `doctor`, `config`, and `completion` subcommands in `TestCLI...` style tests. Extend the existing helpers like `buildCLI` and `runCLI` in `main_test.go` to invoke each subcommand and assert the expected output/behavior, so the new command paths are exercised and protected against regressions.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 83-88: The golangci-lint GitHub Action is using a floating
version, which can make CI unstable when new releases change behavior. Update
the Run golangci-lint step in the workflow to pin the action’s version
explicitly instead of `latest`, and keep the rest of the action configuration
unchanged. Use the `golangci/golangci-lint-action` step to set a specific
released `version` value so upgrades happen deliberately.
- Around line 7-10: The `paths-ignore` configuration in the CI workflow contains
`.github/**`, which prevents CI from running when workflow files are modified,
allowing workflow bugs to slip through to main. Remove the `.github/**` entry
from the `paths-ignore` list, or if you need to ignore certain non-critical
files in the `.github` directory, narrow the pattern to only exclude the
specific subdirectories like `.github/ISSUE_TEMPLATE/**` while keeping the
workflow files in `.github/workflows/**` able to trigger CI.
- Around line 118-127: The dist verification step is searching for the wrong CLI
binary name, so the binary check never matches anything. Update the “Verify
dist/ output” workflow step to look for the actual built artifact name used by
the CLI, deepscanbot, and keep the rest of the dist checks unchanged. Use the
binary naming confirmed in apps/cli/tests/main_test.go and align the find
pattern with the package’s real output.
In @.github/workflows/release.yml:
- Around line 3-7: The release workflow trigger is too broad and currently runs
on every push to main instead of only version tags. Update the workflow’s on
section in the release pipeline to use git tag matching v* (while keeping manual
dispatch if needed), and remove the main branch push trigger so the release job
is only started by versioned releases.
- Around line 55-56: The build job’s “Lint code” step runs golangci-lint
directly even though it is not installed there, so the workflow will fail.
Update the release workflow to install golangci-lint in this job before invoking
golangci-lint run ./..., using the same golangci/golangci-lint-action@v6 setup
pattern already used elsewhere in the workflow. Keep the lint step itself
unchanged once the tool is available.
- Around line 70-111: The create-github-release job is missing the Go toolchain
setup, so the GoReleaser snapshot build in the release workflow can fail. Add a
Go installation step in the job before the Install GoReleaser step, using the
same Node/GoReleaser setup pattern already present in release.yml, so the
goreleaser release --snapshot --clean step has Go available.
- Around line 29-82: The workflow still uses deprecated actions in the build and
release jobs. Update the `actions/checkout` and `actions/setup-node` steps in
the `build` and `create-github-release` jobs to the current major versions used
elsewhere, keeping the existing step names and behavior unchanged so the release
pipeline stays aligned with the newer runner-compatible actions.
In @.gitignore:
- Around line 13-15: The .gitignore entries currently exclude dist/ but not
generated binaries, so add bin/ alongside dist/ in the build artifacts section.
Update the ignore list in .gitignore so install/release-generated outputs are
covered and won’t be committed.
In @.goreleaser.yml:
- Around line 34-38: The GoReleaser ldflags are targeting package-level version
variables that the CLI entrypoint does not currently expose. Update
apps/cli/main.go to declare the main.version, main.commit, and main.date
variables used by the ldflags, and change versionCmd to print those values
instead of the hardcoded v1.0.0 string; if you do not want dynamic build
metadata, remove the corresponding ldflags from the GoReleaser config.
In `@apps/cli/main.go`:
- Line 192: The local variable returned by parseKeyValue in scanCmd shadows the
net/url import, so rename it to something like targetURL or startURL. Update the
assignment in the scanCmd Run scope and any later uses in that function to keep
the package name available for future url.Parse calls.
- Around line 175-178: The scan command help example is using Cobra flag syntax
for retries even though the command accepts only key=value arguments. Update the
example text in main.go’s scan command help so the retries option is shown as
retries=3, matching the other examples and the parsing behavior of the scan
command.
- Around line 191-244: The Cobra command in main should use RunE instead of Run
and return errors rather than calling log.Fatalf. Update the command’s Run
function in the main command setup to RunE, replace each fatal branch around
validateStartURL, buildOutputFilename, storage.ReadEntriesFromFile,
c.StartReport, and the final write step with returned errors, and keep the
existing command flow in the same handler so Cobra can handle reporting and exit
behavior consistently.
- Around line 131-140: Update the argument parsing in the CLI loop that handles
args in main.go so URLs are recognized before treating values as key=value
options. The current strings.Contains(arg, "=") check in the parsing logic can
misclassify URLs with query strings; instead, detect a starting URL first (for
example by checking for a URL scheme or otherwise prioritizing url assignment),
and only pass true option arguments to applyScanOption. Keep the existing
behavior for scan options, but ensure a URL like https://...?...=... is stored
in url rather than split.
- Around line 275-288: The completionCmd Run handler currently ignores the
selected shell and does not generate anything; update completionCmd in main.go
to invoke Cobra’s built-in completion generators based on the args value instead
of discarding shell. Use the existing completionCmd and cmd.Run context to route
bash, zsh, fish, and powershell to the appropriate Cobra generation method, and
write the generated script to stdout.
In `@apps/docs/docs/contribution-guide/how-to-contribute.mdx`:
- Around line 99-104: The contributor guide’s binary verification step points to
a missing script, so update the verification command in how-to-contribute.mdx to
use the existing prepublish check flow instead of the absent verify-binary
helper. Adjust the “Verify binaries” instruction near the build section to
reference node scripts/prepublish-check.js, or add the missing helper if you
want to preserve the current command, and make sure the documented step name
matches the actual script used.
In `@go.mod`:
- Line 8: Upgrade the `golang.org/x/net` dependency in `go.mod` to at least
`v0.55.0` and refresh `go.sum` accordingly. Update the module entry for
`golang.org/x/net`, then run `go mod tidy` so all checksum entries are
regenerated consistently for the `html` crawler usage.
In `@package.json`:
- Line 52: The package.json prepare script currently runs lefthook install for
all npm consumers, which can break installs when lefthook is unavailable. Update
the package manifest so this hook is either removed or guarded to run only in a
contributor/development context, and keep the change centered on the prepare
script entry in package.json.
In `@README.md`:
- Line 5: The npm badge in README.md points to a different package slug than the
published name used elsewhere, so align the badge target with the package name
referenced by the install commands and package metadata. Update the badge URL in
the README’s npm version badge to use the same package identifier as
package.json and the documented install command, or intentionally rename the
package everywhere if that slug is meant to be the canonical one. Use the badge
markup near the top of README.md as the location to fix.
- Around line 261-275: The snapshot release instructions reference helper
scripts that are missing, so add or restore the expected scripts used by the
release flow. Make sure the `scripts/copy-to-npm.sh` and
`scripts/sync-version.js` entries exist and match what `README.md` and the
release workflow invoke, or update those references to the correct existing
helpers such as `scripts/prepublish-check.js` if that is the intended
replacement.
In `@scripts/prepublish-check.js`:
- Around line 42-48: The EXPECTED_PLATFORMS entries in prepublish-check.js are
aligned with the GoReleaser matrix, but the example values in the
EXPECTED_PLATFORMS array use hardcoded dist directory names that may drift from
GoReleaser naming. Update the surrounding code near EXPECTED_PLATFORMS and
findBinaryDirectory to clearly mark these example paths as illustrative only, so
future readers know the matching logic relies on includes/fuzzy matching rather
than exact directory names.
- Line 197: The binary help invocation in prepublish-check should avoid shell
execution: replace the `childProcess.execSync` call used for `binPath` with
`childProcess.execFileSync` so the executable is run directly without shell
interpretation. Update the `execFileSync` call in the same prepublish-check flow
to pass `binPath` as the file and `--help` as the argument, preserving the
ignored stdio behavior.
---
Outside diff comments:
In `@apps/cli/tests/main_test.go`:
- Around line 13-77: The CLI test suite currently covers scan and help only; add
unit coverage for the new `version`, `doctor`, `config`, and `completion`
subcommands in `TestCLI...` style tests. Extend the existing helpers like
`buildCLI` and `runCLI` in `main_test.go` to invoke each subcommand and assert
the expected output/behavior, so the new command paths are exercised and
protected against regressions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 70d5c9ea-fcc9-4dd8-aba9-3a0cee19a944
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (19)
.github/workflows/ci.yml.github/workflows/release-docs.yml.github/workflows/release.yml.gitignore.golangci.yml.goreleaser.ymlREADME.mdapps/cli/main.goapps/cli/tests/main_test.goapps/docs/docs/contribution-guide/how-to-contribute.mdxapps/docs/docs/guide/features.mdxapps/docs/docs/guide/usage.mdxapps/docs/docs/installation.mdxapps/docs/docs/introduction.mdxapps/docs/src/pages/index.tsxgo.modpackage.jsonpostinstall.jsscripts/prepublish-check.js
📜 Review details
⚠️ CI failures not shown inline (3)
GitHub Actions: CI / Build and Test: Release from Development
Conclusion: failure
##[group]Run go test -v -race -count=1 -coverprofile=coverage.out ./...
�[36;1mgo test -v -race -count=1 -coverprofile=coverage.out ./...�[0m
shell: /usr/bin/bash -e {0}
env:
GO_VERSION: 1.22.4
NODE_VERSION: 20
GORELEASER_VERSION: ~> v2
##[endgroup]
github.com/mindfiredigital/DeepScanBot/apps/cli coverage: 0.0% of statements
=== RUN TestCrawlerStartReturnsResultsWithoutWritingFiles
level=INFO msg="Starting crawl: url=http://127.0.0.1:35189 max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:35189 (depth=0)"
level=INFO msg="Crawled http://127.0.0.1:35189 [status=200] [result=passed]"
level=INFO msg="Crawl finished: url=http://127.0.0.1:35189 total=1 passed=1 failed=0 skipped=0 duration=2ms"
--- PASS: TestCrawlerStartReturnsResultsWithoutWritingFiles (0.00s)
=== RUN TestCrawlerRespectsRobots
level=INFO msg="Starting crawl: url=http://127.0.0.1:36137/public max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:36137/public (depth=0)"
level=INFO msg="Crawled http://127.0.0.1:36137/public [status=200] [result=passed]"
level=INFO msg="Crawl finished: url=http://127.0.0.1:36137/public total=1 passed=1 failed=0 skipped=0 duration=2ms"
level=INFO msg="Starting crawl: url=http://127.0.0.1:36137/private max-depth=0 concurrency=1 per-host-concurrency=1 retries=0 delay=0s sitemap=false resumed=0"
level=INFO msg="Crawling http://127.0.0.1:36137/private (depth=0)"
level=INFO msg="Skipping http://127.0.0.1:36137/private because robots.txt disallows it"
level=INFO msg="Crawl finished: url=http://127.0.0.1:36137/private total=1 passed=0 failed=0 skipped=1 duration=1ms"
--- PASS: TestCrawlerRespectsRobots (0.00s)
=== RUN TestCrawlerProcessesDiscoveredLinks
level=INFO msg="Starting crawl: url=http://127.0.0.1:42433 max-depth=1 concurrency=1 per-host-concurrency=1 retries=0 delay=0s ...
GitHub Actions: CI / Build and Test: Release from Development
Conclusion: failure
##[group]run golangci-lint
Running [/home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config path] in [/home/runner/work/DeepScanBot/DeepScanBot] ...
Running [/home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config verify] in [/home/runner/work/DeepScanBot/DeepScanBot] ...
##[endgroup]
##[error]Failed to run: Error: Command failed: /home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config verify
GitHub Actions: CI / 6_Build and Test.txt: Release from Development
Conclusion: failure
##[group]run golangci-lint
Running [/home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config path] in [/home/runner/work/DeepScanBot/DeepScanBot] ...
Running [/home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config verify] in [/home/runner/work/DeepScanBot/DeepScanBot] ...
##[endgroup]
##[error]Failed to run: Error: Command failed: /home/runner/golangci-lint-1.64.8-linux-amd64/golangci-lint config verify
🧰 Additional context used
📓 Path-based instructions (12)
**
⚙️ CodeRabbit configuration file
**: # Contributing to DeepScanBotThank you for your interest in contributing to DeepScanBot! This document provides guidelines and instructions for contributing to this project.
Table of Contents
- Code of Conduct
- Getting Started
- Development Setup
- How to Contribute
- Pull Request Process
- Coding Standards
- Testing
- Reporting Bugs
- Suggesting Features
Code of Conduct
By participating in this project, you agree to abide by the CODE_OF_CONDUCT.md. Please be respectful and constructive in all interactions.
Getting Started
- Fork the repository on GitHub
- Clone your fork locally:
git clone https://github.com/YOUR_USERNAME/DeepScanBot.git cd DeepScanBot- Add the upstream remote:
git remote add upstream https://github.com/mindfiredigital/DeepScanBot.gitDevelopment Setup
Prerequisites
- Go (version 1.21 or higher)
- Git
Installation
Install dependencies:
go mod downloadInstall development tools:
go install mvdan.cc/gofumpt@latest go install github.com/daixiang0/gci@latest go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest go install github.com/evilmartians/lefthook@latestInstall git hooks:
lefthook installBuild the project:
go build -o deepscanbot .Verify the installation:
go run ./apps/cli -hProject Structure
DeepScanBot/ ├── crawler/ # Core crawling logic and orchestration ├── fetcher/ # HTTP fetching and content-type filtering ├── parser/ # HTML parsing and link extraction ├── storage/ # Data storage and report generation ├── logger/ # Logging utilitie...
Files:
apps/docs/src/pages/index.tsxpackage.jsongo.modpostinstall.jsscripts/prepublish-check.jsapps/docs/docs/installation.mdxapps/cli/tests/main_test.goapps/docs/docs/introduction.mdxREADME.mdapps/docs/docs/contribution-guide/how-to-contribute.mdxapps/docs/docs/guide/features.mdxapps/docs/docs/guide/usage.mdxapps/cli/main.go
package.json
📄 CodeRabbit inference engine (README.md)
The npm package configuration in
package.jsonshould support version updates from release tags and npm publishing as part of the release flow.
Files:
package.json
go.mod
⚙️ CodeRabbit configuration file
go.mod: - Module path must match the repository URL
- Dependencies should be pinned to specific versions
- No indirect dependencies unless necessary
- Go version should be specified
Files:
go.mod
postinstall.js
📄 CodeRabbit inference engine (README.md)
postinstall.jsshould install the correct binary for the current platform during npm installation.
Files:
postinstall.js
.goreleaser.yml
📄 CodeRabbit inference engine (README.md)
The GoReleaser configuration in
.goreleaser.ymlshould define the build and release packaging process used to produce binaries and checksums for distribution.
Files:
.goreleaser.yml
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.go: Follow Effective Go guidelines for all Go code.
Format Go code withgofumpt(strict gofmt).
Usegoimportsorgcito manage imports in Go files.
Follow the Go Code Review Comments in all Go code.
Write unit tests for new functionality.
Files:
apps/cli/tests/main_test.goapps/cli/main.go
**/*_test.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*_test.go: Place test files alongside the code they test, such ascrawler_test.gonext tocrawler.go.
Use table-driven tests where appropriate.
Test both success and failure cases.
Mock external dependencies when necessary.
Files:
apps/cli/tests/main_test.go
⚙️ CodeRabbit configuration file
**/*_test.go: ## Test Review Checklist
- Tests should be independent and repeatable
- Use table-driven tests for multiple scenarios
- Mock external dependencies (HTTP, file system)
- Test both success and failure paths
- Use meaningful test names that describe the scenario
Files:
apps/cli/tests/main_test.go
apps/cli/**/*.go
📄 CodeRabbit inference engine (README.md)
apps/cli/**/*.go: Thedeepscanbot scan <url> [key=value ...]CLI implementation should acceptkey=valueoptions after the URL and support the documented scan options and defaults such asdepth,timeout,proxy,json,size,disable-redirects,show-source,insecure,unique,concurrency,host-concurrency,content-types,output,ignore-robots,cross-domain,retries,retry-backoff,delay,sitemap, andresume.
The CLI should expose theversion,doctor,config,completion, andhelpcommands with the documented behavior.
Thecompletioncommand should generate shell completion scripts.
Thedoctorcommand should verify installation and environment readiness.
Files:
apps/cli/tests/main_test.goapps/cli/main.go
.github/workflows/ci.yml
📄 CodeRabbit inference engine (README.md)
The CI workflow in
.github/workflows/ci.ymlshould run on every push and pull request tomainand perform Go builds, race-detection tests,golangci-lint, Go formatting checks, GoReleaser config verification, cross-platform build checks, and npm package validation.
Files:
.github/workflows/ci.yml
.github/workflows/release.yml
📄 CodeRabbit inference engine (README.md)
The release workflow in
.github/workflows/release.ymlshould be triggered by Git tags matchingv*and handle GoReleaser builds, copying binaries into the npm package, updatingpackage.jsonversion from the tag, generating SHA256 checksums, verifying binaries, publishing to npm, and uploading release artifacts.
Files:
.github/workflows/release.yml
.golangci.yml
⚙️ CodeRabbit configuration file
.golangci.yml: - Linter configuration should be comprehensive but not overwhelming
- Enable security linters (gosec)
- Enable performance linters (prealloc, gocritic)
- Document why linters are disabled
Files:
.golangci.yml
apps/cli/*.go
⚙️ CodeRabbit configuration file
apps/cli/*.go: ## CLI Application Review Checklist
- Command-line flags should have clear help text
- Validate all user input before processing
- Use structured logging (not fmt.Println)
- Handle signals gracefully (SIGINT, SIGTERM)
- Exit with appropriate status codes (0 for success, non-zero for failure)
Files:
apps/cli/main.go
🪛 actionlint (1.7.12)
.github/workflows/release.yml
[error] 29-29: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 38-38: the runner of "actions/setup-node@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 79-79: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 82-82: the runner of "actions/setup-node@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🪛 ast-grep (0.44.1)
postinstall.js
[error] 59-59: An archive entry path (e.g. entry.path / entry.fileName / header.name) is joined to an output directory without validating that the resolved path stays inside that directory. A malicious archive can use "../" sequences to escape the extraction directory and overwrite arbitrary files (Zip Slip). Resolve the path and verify it starts with the normalized output directory, or strip traversal with path.basename, before writing the entry.
Context: path.join(DIST_DIR, entry.name)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(zip-slip-archive-extraction-javascript)
scripts/prepublish-check.js
[error] 73-73: An archive entry path (e.g. entry.path / entry.fileName / header.name) is joined to an output directory without validating that the resolved path stays inside that directory. A malicious archive can use "../" sequences to escape the extraction directory and overwrite arbitrary files (Zip Slip). Resolve the path and verify it starts with the normalized output directory, or strip traversal with path.basename, before writing the entry.
Context: path.join(DIST_DIR, entry.name)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(zip-slip-archive-extraction-javascript)
[warning] 25-25: Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: require("child_process")
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(detect-child-process)
[warning] 95-95: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(PROJECT_ROOT, "package.json"), "utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🪛 LanguageTool
README.md
[uncategorized] ~215-~215: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...nges - MINOR (x.1.x): New features, backward compatible - PATCH (x.x.1): Bug fixes, backwar...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🪛 markdownlint-cli2 (0.22.1)
README.md
[warning] 169-169: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
[warning] 196-196: Ordered list item prefix
Expected: 1; Actual: 2; Style: 1/2/3
(MD029, ol-prefix)
[warning] 204-204: Ordered list item prefix
Expected: 2; Actual: 3; Style: 1/2/3
(MD029, ol-prefix)
[warning] 279-279: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🪛 OpenGrep (1.23.0)
scripts/prepublish-check.js
[ERROR] 197-197: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.
(coderabbit.command-injection.exec-js)
🪛 OSV Scanner (2.4.0)
go.mod
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Non-linear parsing of case-insensitive content in golang.org/x/net/html
(GO-2024-3333)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: HTTP Proxy bypass using IPv6 Zone IDs in golang.org/x/net
(GO-2025-3503)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Incorrect Neutralization of Input During Web Page Generation in x/net in golang.org/x/net
(GO-2025-3595)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Quadratic parsing complexity in golang.org/x/net/html
(GO-2026-4440)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Infinite parsing loop in golang.org/x/net
(GO-2026-4441)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Infinite loop in HTTP/2 transport when given bad SETTINGS_MAX_FRAME_SIZE in net/http/internal/http2 in golang.org/x/net
(GO-2026-4918)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking incorrect handling of namespaced elements in foreign content in golang.org/x/net/html
(GO-2026-5025)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking failure to reject ASCII-only Punycode-encoded labels in golang.org/x/net/idna
(GO-2026-5026)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking incorrect handling of HTML elements in foreign content in golang.org/x/net/html
(GO-2026-5027)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking denial of service when parsing arbitrary HTML in golang.org/x/net/html
(GO-2026-5028)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking incorrect handling of character references in DOCTYPE nodes in golang.org/x/net/html
(GO-2026-5029)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Invoking duplicate attributes can cause XSS in golang.org/x/net/html
(GO-2026-5030)
[MEDIUM] 8-8: golang.org/x/net 0.26.0: Go Net HTML parser is vulnerable to denial of service
[MEDIUM] 8-8: golang.org/x/net 0.26.0: HTTP Proxy bypass using IPv6 Zone IDs in golang.org/x/net
[MEDIUM] 8-8: golang.org/x/net 0.26.0: golang.org/x/net vulnerable to Cross-site Scripting
🪛 zizmor (1.26.1)
.github/workflows/ci.yml
[warning] 33-36: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 97-98: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 140-141: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 159-160: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 20-20: overly broad permissions (excessive-permissions): checks: write is overly broad at the workflow level
(excessive-permissions)
[error] 21-21: overly broad permissions (excessive-permissions): pull-requests: write is overly broad at the workflow level
(excessive-permissions)
[warning] 151-151: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
[warning] 151-151: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
[error] 34-34: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 39-39: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 45-45: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 67-67: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 84-84: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 98-98: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 101-101: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 107-107: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 141-141: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 144-144: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 160-160: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 163-163: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[warning] 20-20: permissions without explanatory comments (undocumented-permissions): needs an explanatory comment
(undocumented-permissions)
[error] 101-101: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): this step
(cache-poisoning)
.github/workflows/release.yml
[warning] 28-29: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 78-79: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 10-10: overly broad permissions (excessive-permissions): contents: write is overly broad at the workflow level
(excessive-permissions)
[error] 11-11: overly broad permissions (excessive-permissions): packages: write is overly broad at the workflow level
(excessive-permissions)
[error] 12-12: overly broad permissions (excessive-permissions): id-token: write is overly broad at the workflow level
(excessive-permissions)
[error] 29-29: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 32-32: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 38-38: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 43-43: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 79-79: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 82-82: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 87-87: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[warning] 10-10: permissions without explanatory comments (undocumented-permissions): needs an explanatory comment
(undocumented-permissions)
[warning] 25-25: permissions without explanatory comments (undocumented-permissions): needs an explanatory comment
(undocumented-permissions)
[warning] 75-75: permissions without explanatory comments (undocumented-permissions): needs an explanatory comment
(undocumented-permissions)
[error] 82-82: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): enables caching by default
(cache-poisoning)
[warning] 3-7: insufficient job-level concurrency limits (concurrency-limits): workflow is missing concurrency setting
(concurrency-limits)
🔇 Additional comments (14)
.github/workflows/release-docs.yml (1)
11-11: LGTM!Also applies to: 70-70
.golangci.yml (1)
6-78: LGTM!postinstall.js (1)
1-123: LGTM!.goreleaser.yml (1)
1-81: LGTM on the GoReleaser configuration overall. Clean v2 format, correct platform matrix with windows/arm64 exclusion, binary archive format aligns with the postinstall.js expectations, anddist: distis consistent across all packaging scripts.package.json (1)
35-41: 🗄️ Data Integrity & IntegrationNo change needed
LICENSE.mdis present, andscripts/prepublish-check.jsalready requires it, so thefilesentry and publish guard are aligned.> Likely an incorrect or invalid review comment.apps/docs/docs/guide/features.mdx (1)
195-216: 📐 Maintainability & Code Quality | ⚡ Quick win
host-concurrencyrow still missing the "0 = uses -concurrency" fallback note.Same gap flagged previously; the
-concurrencyrow keeps its "(0 = CPU)" clarification buthost-concurrencydoesn't explain its fallback semantics.📝 Proposed fix
-| `host-concurrency` | `0` | Max concurrent requests per host | +| `host-concurrency` | `0` | Max concurrent requests per host (0 = uses concurrency) |apps/docs/docs/contribution-guide/how-to-contribute.mdx (1)
106-118: 🗄️ Data Integrity & Integration | 🏗️ Heavy liftPackage name mismatch:
deep-scan-botvsdeepscanbot.Uses
deep-scan-bot-*.tgz/@mindfiredigital/deep-scan-bothere, butinstallation.mdxand most ofREADME.md's Quick Start use@mindfiredigital/deepscanbot. Confirm the actual published package name inpackage.jsonand align all docs (README badges/CTAs also use both forms).apps/docs/docs/introduction.mdx (2)
7-44: LGTM!
184-217: LGTM!apps/docs/src/pages/index.tsx (2)
34-37: 🗄️ Data Integrity & Integration | ⚡ Quick winTerminal demo still shows the deprecated single-dash flag CLI syntax.
deepscanbot -url https://example.com -depth 2 -json -output scandoesn't match the new Cobrascan <url> key=valuecontract documented in README/usage.mdx/features.mdx (e.g.,deepscanbot scan https://example.com depth=3 json=true). This is the site's landing page — a stale example here undermines the rest of the doc rewrite.📝 Proposed fix
- <span className={styles.terminalCommand}>deepscanbot -url https://example.com -depth 2 -json -output scan</span> + <span className={styles.terminalCommand}>deepscanbot scan https://example.com depth=2 json=true output=scan</span>
135-137: LGTM!README.md (3)
169-169: 📐 Maintainability & Code Quality | 💤 Low valueFenced block missing a language hint (MD040).
📝 Proposed fix
-``` +```text Git Tag (v1.0.0)Source: Linters/SAST tools
279-279: 📐 Maintainability & Code Quality | 💤 Low valueFenced block missing a language hint (MD040).
📝 Proposed fix
-``` +```text project/Source: Linters/SAST tools
189-208: 📐 Maintainability & Code Quality | 💤 Low valueOrdered list prefixes flagged by markdownlint (MD029).
Lines 196/204 numbering doesn't match the configured
ol-prefixstyle (likely expects1.for every item under the lint config). Low priority but a quick fix if you want CI markdownlint green.Source: Linters/SAST tools
| paths-ignore: | ||
| - "docs/**" | ||
| - "**.md" | ||
| - ".github/**" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
paths-ignore includes .github/**, so changes to workflow files won't trigger CI.
This means workflow bugs (like the my-cli* issue above) won't be caught until they reach main. Consider removing .github/** from paths-ignore or narrowing it to exclude only .github/ISSUE_TEMPLATE/**.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 7 - 10, The `paths-ignore`
configuration in the CI workflow contains `.github/**`, which prevents CI from
running when workflow files are modified, allowing workflow bugs to slip through
to main. Remove the `.github/**` entry from the `paths-ignore` list, or if you
need to ignore certain non-critical files in the `.github` directory, narrow the
pattern to only exclude the specific subdirectories like
`.github/ISSUE_TEMPLATE/**` while keeping the workflow files in
`.github/workflows/**` able to trigger CI.
| - name: Run golangci-lint | ||
| uses: golangci/golangci-lint-action@v6 | ||
| with: | ||
| version: latest | ||
| args: --timeout=5m --out-format=colored-line-number | ||
| skip-cache: false |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
Pin golangci-lint version instead of using latest to avoid breaking on new releases.
Using version: latest means a new golangci-lint release with stricter defaults could break CI without any code change. Pin to a specific version (e.g., v1.62.0) and update deliberately.
🧰 Tools
🪛 zizmor (1.26.1)
[error] 84-84: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 83 - 88, The golangci-lint GitHub
Action is using a floating version, which can make CI unstable when new releases
change behavior. Update the Run golangci-lint step in the workflow to pin the
action’s version explicitly instead of `latest`, and keep the rest of the action
configuration unchanged. Use the `golangci/golangci-lint-action` step to set a
specific released `version` value so upgrades happen deliberately.
| - name: Verify dist/ output | ||
| run: | | ||
| golangci-lint run --timeout 5m ./... | ||
| echo "✅ Lint passed" | ||
| echo "=== dist/ contents ===" | ||
| ls -la dist/ | ||
| echo "" | ||
| echo "=== Binary files ===" | ||
| find dist/ -type f -name "my-cli*" -exec file {} \; | ||
| echo "" | ||
| echo "=== Checksums ===" | ||
| cat dist/checksums.txt 2>/dev/null || echo "No checksums file found" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
find dist/ -type f -name "my-cli*" uses the wrong binary name — should be deepscanbot*.
The CLI binary is built as deepscanbot (confirmed by test setup at apps/cli/tests/main_test.go:87 and README build instructions). This find command will match nothing, making the binary verification step a no-op.
🐛 Proposed fix
- find dist/ -type f -name "my-cli*" -exec file {} \;
+ find dist/ -type f -name "deepscanbot*" -exec file {} \;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Verify dist/ output | |
| run: | | |
| golangci-lint run --timeout 5m ./... | |
| echo "✅ Lint passed" | |
| echo "=== dist/ contents ===" | |
| ls -la dist/ | |
| echo "" | |
| echo "=== Binary files ===" | |
| find dist/ -type f -name "my-cli*" -exec file {} \; | |
| echo "" | |
| echo "=== Checksums ===" | |
| cat dist/checksums.txt 2>/dev/null || echo "No checksums file found" | |
| - name: Verify dist/ output | |
| run: | | |
| echo "=== dist/ contents ===" | |
| ls -la dist/ | |
| echo "" | |
| echo "=== Binary files ===" | |
| find dist/ -type f -name "deepscanbot*" -exec file {} \; | |
| echo "" | |
| echo "=== Checksums ===" | |
| cat dist/checksums.txt 2>/dev/null || echo "No checksums file found" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 118 - 127, The dist verification step
is searching for the wrong CLI binary name, so the binary check never matches
anything. Update the “Verify dist/ output” workflow step to look for the actual
built artifact name used by the CLI, deepscanbot, and keep the rest of the dist
checks unchanged. Use the binary naming confirmed in apps/cli/tests/main_test.go
and align the find pattern with the package’s real output.
| on: | ||
| push: | ||
| tags: | ||
| - 'v*' | ||
| branches: | ||
| - main | ||
| workflow_dispatch: |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Workflow triggers on every push to main — should trigger on version tags per coding guidelines.
Per coding guidelines, the release workflow should be triggered by Git tags matching v*. Triggering on every push to main will attempt a release on every merge, which is almost certainly not intended for a release pipeline.
📝 Proposed fix: trigger on tags
on:
- push:
- branches:
- - main
+ push:
+ tags:
+ - "v*"
workflow_dispatch:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| on: | |
| push: | |
| tags: | |
| - 'v*' | |
| branches: | |
| - main | |
| workflow_dispatch: | |
| on: | |
| push: | |
| tags: | |
| - "v*" | |
| workflow_dispatch: |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 3-7: insufficient job-level concurrency limits (concurrency-limits): workflow is missing concurrency setting
(concurrency-limits)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/release.yml around lines 3 - 7, The release workflow
trigger is too broad and currently runs on every push to main instead of only
version tags. Update the workflow’s on section in the release pipeline to use
git tag matching v* (while keeping manual dispatch if needed), and remove the
main branch push trigger so the release job is only started by versioned
releases.
Source: Coding guidelines
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: '1.22' | ||
| check-latest: true | ||
| go-version: ${{ env.GO_VERSION }} | ||
| cache: true | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@v3 | ||
| with: | ||
| node-version: ${{ env.NODE_VERSION }} | ||
|
|
||
| - name: Cache Go modules | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: | | ||
| ~/go/pkg/mod | ||
| ~/.cache/go-build | ||
| key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-go- | ||
|
|
||
| - name: Install dependencies | ||
| run: go mod download | ||
| run: npm ci | ||
|
|
||
| - name: Lint code | ||
| run: golangci-lint run ./... | ||
|
|
||
| - name: Run tests | ||
| run: go test -v -count=1 ./... | ||
| run: go test ./... | ||
|
|
||
| - name: Build binaries | ||
| - name: Restore changes in json | ||
| run: | | ||
| mkdir -p dist | ||
| GOOS=linux GOARCH=amd64 go build -o dist/deepscanbot-linux-amd64 ./apps/cli | ||
| GOOS=linux GOARCH=arm64 go build -o dist/deepscanbot-linux-arm64 ./apps/cli | ||
| GOOS=darwin GOARCH=amd64 go build -o dist/deepscanbot-darwin-amd64 ./apps/cli | ||
| GOOS=darwin GOARCH=arm64 go build -o dist/deepscanbot-darwin-arm64 ./apps/cli | ||
| GOOS=windows GOARCH=amd64 go build -o dist/deepscanbot-windows-amd64.exe ./apps/cli | ||
| echo "✅ Binaries built" | ||
|
|
||
| - name: Generate checksums | ||
| git restore package-lock.json 2>/dev/null || true | ||
|
|
||
| - name: Set Git user name and email | ||
| run: | | ||
| cd dist | ||
| sha256sum * > checksums.txt | ||
| echo "✅ Checksums generated" | ||
| git config --local user.email "github-actions@github.com" | ||
| git config --local user.name "GitHub Actions" | ||
|
|
||
| create-github-release: | ||
| name: Create GitHub release document and publish to npm | ||
| runs-on: ubuntu-latest | ||
| needs: build | ||
| permissions: | ||
| contents: write | ||
| packages: write | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Create Release | ||
| uses: softprops/action-gh-release@v2 | ||
| - name: Set up Node.js | ||
| uses: actions/setup-node@v3 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Deprecated @v3 actions — update to current major versions.
actions/checkout@v3 and actions/setup-node@v3 are deprecated. The CI workflow already uses @v4/@v5. Align these to avoid runner compatibility issues.
📝 Proposed fix
- - name: Checkout repository
- uses: actions/checkout@v3
+ - name: Checkout repository
+ uses: actions/checkout@v4
- - name: Set up Node.js
- uses: actions/setup-node@v3
+ - name: Set up Node.js
+ uses: actions/setup-node@v4🧰 Tools
🪛 actionlint (1.7.12)
[error] 29-29: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 38-38: the runner of "actions/setup-node@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 79-79: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
[error] 82-82: the runner of "actions/setup-node@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🪛 zizmor (1.26.1)
[warning] 78-79: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 29-29: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 32-32: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 38-38: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 43-43: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 79-79: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 82-82: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[warning] 75-75: permissions without explanatory comments (undocumented-permissions): needs an explanatory comment
(undocumented-permissions)
[error] 82-82: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): enables caching by default
(cache-poisoning)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/release.yml around lines 29 - 82, The workflow still uses
deprecated actions in the build and release jobs. Update the `actions/checkout`
and `actions/setup-node` steps in the `build` and `create-github-release` jobs
to the current major versions used elsewhere, keeping the existing step names
and behavior unchanged so the release pipeline stays aligned with the newer
runner-compatible actions.
| "verify": "node scripts/prepublish-check.js", | ||
| "test": "go test ./...", | ||
| "pack": "npm pack", | ||
| "prepare": "lefthook install", |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Remove or guard the prepare script for npm consumers.
The prepare script runs lefthook install on npm install for end-users who install this package globally, not just contributors. Most consumers won't have lefthook installed, causing a post-installation error. npm runs prepare before prepublishOnly and also on local npm install without arguments.
Guard it so it only runs in a development context:
🔧 Proposed fix
- "prepare": "lefthook install",
+ "prepare": "test -d .git && lefthook install || true",Or remove it entirely and document lefthook install as a manual contributor step.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "prepare": "lefthook install", | |
| "prepare": "test -d .git && lefthook install || true", |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@package.json` at line 52, The package.json prepare script currently runs
lefthook install for all npm consumers, which can break installs when lefthook
is unavailable. Update the package manifest so this hook is either removed or
guarded to run only in a contributor/development context, and keep the change
centered on the prepare script entry in package.json.
| DeepScanBot is a customizable, feature-rich web crawler written in Go. It recursively crawls web pages, respects robots.txt, handles rate-limiting, supports retries, sitemap discovery, resume mode, and produces detailed JSON or text reports. | ||
| [](https://github.com/mindfiredigital/DeepScanBot/actions/workflows/ci.yml) | ||
| [](https://github.com/mindfiredigital/DeepScanBot/actions/workflows/release.yml) | ||
| [](https://www.npmjs.com/package/@mindfiredigital/deep-scan-bot) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd package.json -d 2 -x cat {}Repository: mindfiredigital/DeepScanBot
Length of output: 1994
Align the npm badge with the published package name. README.md:5 The badge links to @mindfiredigital/deep-scan-bot, while package.json and the install commands use @mindfiredigital/deepscanbot. Update the badge URL, or rename the package if that slug is intentional, so copy-paste paths stay consistent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 5, The npm badge in README.md points to a different
package slug than the published name used elsewhere, so align the badge target
with the package name referenced by the install commands and package metadata.
Update the badge URL in the README’s npm version badge to use the same package
identifier as package.json and the documented install command, or intentionally
rename the package everywhere if that slug is meant to be the canonical one. Use
the badge markup near the top of README.md as the location to fix.
| ### Snapshot Releases | ||
|
|
||
| #### 6. Crawl Goodreads with Rate-Limit Handling | ||
| For testing purposes, you can create snapshot releases: | ||
|
|
||
| ```bash | ||
| go run ./apps/cli -url https://www.goodreads.com -depth 2 -delay 2s -retries 5 -retry-backoff 2s -concurrency 2 -host-concurrency 1 -json -output goodreads_results | ||
| # Create a snapshot build | ||
| goreleaser release --snapshot --clean | ||
|
|
||
| # Copy binaries and update version | ||
| bash scripts/copy-to-npm.sh | ||
| VERSION=0.0.0-snapshot node scripts/sync-version.js | ||
|
|
||
| # Dry run publish | ||
| npm publish --dry-run | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd 'copy-to-npm.sh|sync-version.js' scriptsRepository: mindfiredigital/DeepScanBot
Length of output: 165
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Top-level files:\n'
git ls-files | sed -n '1,200p' | rg '^(README\.md|scripts/|package\.json|go\.mod|goreleaser|\.github/)'
printf '\nScript files under scripts/:\n'
fd -a . scripts
printf '\nExact helper names anywhere in repo:\n'
rg -n --hidden --glob '!**/.git/**' 'copy-to-npm\.sh|sync-version\.js|prepublish-check\.js|postinstall\.js|snapshot release|goreleaser release --snapshot' .Repository: mindfiredigital/DeepScanBot
Length of output: 2811
Add the missing snapshot-release helpers
README.md points to bash scripts/copy-to-npm.sh and node scripts/sync-version.js, but neither file exists under scripts/ (only scripts/prepublish-check.js does). The release workflow also references scripts/copy-to-npm.sh, so this snapshot-release path is broken as written.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 261 - 275, The snapshot release instructions
reference helper scripts that are missing, so add or restore the expected
scripts used by the release flow. Make sure the `scripts/copy-to-npm.sh` and
`scripts/sync-version.js` entries exist and match what `README.md` and the
release workflow invoke, or update those references to the correct existing
helpers such as `scripts/prepublish-check.js` if that is the intended
replacement.
| const EXPECTED_PLATFORMS = [ | ||
| { os: "linux", arch: "amd64", example: "dist/deepscanbot_linux_amd64_v1/" }, | ||
| { os: "linux", arch: "arm64", example: "dist/deepscanbot_linux_arm64_v8.0/" }, | ||
| { os: "darwin", arch: "amd64", example: "dist/deepscanbot_darwin_amd64_v1/" }, | ||
| { os: "darwin", arch: "arm64", example: "dist/deepscanbot_darwin_arm64_v8.0/" }, | ||
| { os: "windows", arch: "amd64", example: "dist/deepscanbot_windows_amd64_v1/" }, | ||
| ]; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
EXPECTED_PLATFORMS should match GoReleaser output exactly.
The five platforms listed here (linux/amd64, linux/arm64, darwin/amd64, darwin/arm64, windows/amd64) correctly match the .goreleaser.yml build matrix which ignores windows/arm64. Good alignment.
However, the example field uses hardcoded directory names like dist/deepscanbot_linux_arm64_v8.0/ which include GoReleaser's ARM version suffix. If GoReleaser changes its naming convention, these examples become misleading. Since findBinaryDirectory uses fuzzy includes matching, the examples are only informational — but consider adding a comment noting they are illustrative.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/prepublish-check.js` around lines 42 - 48, The EXPECTED_PLATFORMS
entries in prepublish-check.js are aligned with the GoReleaser matrix, but the
example values in the EXPECTED_PLATFORMS array use hardcoded dist directory
names that may drift from GoReleaser naming. Update the surrounding code near
EXPECTED_PLATFORMS and findBinaryDirectory to clearly mark these example paths
as illustrative only, so future readers know the matching logic relies on
includes/fuzzy matching rather than exact directory names.
| // Execution check (Native platform only) | ||
| if (platform.os === currentOsStr && platform.arch === currentArchStr) { | ||
| try { | ||
| childProcess.execSync(`"${binPath}" --help`, { stdio: "ignore" }); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Use execFileSync instead of execSync for binary execution.
childProcess.execSync with a string argument passed to a shell is a command-injection pattern. Although binPath is derived from GoReleaser output (not direct user input), a directory name containing shell metacharacters could cause unexpected behavior. Use execFileSync with an argument array to avoid shell interpretation entirely.
🔒 Proposed fix
- childProcess.execSync(`"${binPath}" --help`, { stdio: "ignore" });
+ childProcess.execFileSync(binPath, ["--help"], { stdio: "ignore" });📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| childProcess.execSync(`"${binPath}" --help`, { stdio: "ignore" }); | |
| childProcess.execFileSync(binPath, ["--help"], { stdio: "ignore" }); |
🧰 Tools
🪛 OpenGrep (1.23.0)
[ERROR] 197-197: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.
(coderabbit.command-injection.exec-js)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/prepublish-check.js` at line 197, The binary help invocation in
prepublish-check should avoid shell execution: replace the
`childProcess.execSync` call used for `binPath` with `childProcess.execFileSync`
so the executable is run directly without shell interpretation. Update the
`execFileSync` call in the same prepublish-check flow to pass `binPath` as the
file and `--help` as the argument, preserving the ignored stdio behavior.
Source: Linters/SAST tools
3 participants
Release from Development
Package structure or API changes
cobrawith ascansubcommand andkey=valueoption parsing (plus relatedversion,doctor,config,completioncommands).github.com/spf13/cobratogo.modand introduced new npm package packaging metadata and publish tooling for@mindfiredigital/deepscanbot(includingpostinstall.jsandbin/deepscanbotentry).Bug fixes
Tooling / config changes
mainpushes / PRs tomain, with revised Go/Node/goreleaser checks, race tests + coverage, formatting (gofmt),golangci-lint, and vet.semantic-release..goreleaser.yml, updated.golangci.yml, updated.gitignoreto includedist/.scripts/prepublish-check.jsto validate package contents, expected dist artifacts, binaries/executability, and basic install/run sanity.Documentation updates
README.mdto be CLI- and npm-centric (commands, scan option reference, install/release/versioning/publishing flow, troubleshooting).Breaking changes
cobrasubcommands (notablyscan) withkey=valuearguments, requiring updated invocation patterns and tests/docs.