27 examples
Command injection
Malicious commands injected via user input fields.
[ FAQ1 ]
What is command injection?
Command injection is a security vulnerability that allows attackers to insert and execute malicious shell commands by manipulating insecure input fields. This typically occurs when user-supplied data is directly concatenated or passed to operating system commands without proper validation or sanitization. Attackers exploit command injection to compromise system integrity, steal sensitive data, or gain unauthorized system access. Common risky practices include using functions like
os.system() in Python or shell execution calls in other languages without strict input control.[ FAQ2 ]
How to prevent command injection
To prevent command injection, always rigorously validate, sanitize, and encode user inputs before passing them to operating system commands. Use secure APIs and libraries designed explicitly to handle command execution safely, avoiding direct concatenation of user inputs into shell commands. Employ safer alternatives to command execution functions, such as parameterized commands or subprocess APIs (e.g., Python's
subprocess.run() with argument lists) that inherently mitigate injection risks. Adopting secure coding standards, regular security audits, and using static analysis tools to detect injection vulnerabilities proactively helps ensure robust application security.diff block
+import { LaunchProps, showHUD, showToast, Toast } from "@raycast/api";
+import { execSync } from "child_process";
+import { checkAdbExists, getAppIdFromParamsOrCache, saveAppIdInCache } from "./utils";
+import Style = Toast.Style;
+
+export default async function uninstall(props: LaunchProps<{ arguments: { appId: string | undefined } }>) {
+ let adbDir: string;
+ try {
+ adbDir = await checkAdbExists();
+ } catch (e) {
+ await showHUD(`${e}`);
+ return;
+ }
+ const appId = getAppIdFromParamsOrCache(props.arguments.appId);
+ if (appId === undefined || appId.trim().length == 0 || !appId.match(".*\\..*")) {
+ await showToast(Style.Failure, `AppId "${appId}" doesn't match com.vendor format`);
+ return;
+ }
+ saveAppIdInCache(appId);
+ await showHUD(`🗑️ Uninstall ${appId}`);
+ execSync(`${adbDir} uninstall ${appId}`);
greptile
logic: appId is not escaped - could allow command injection if malicious input is provided
diff block
+/* eslint-disable @typescript-eslint/no-var-requires */
+/**
+ * Safari Bookmarks Parser Example
+ *
+ * This example demonstrates how to call the pre-compiled Safari bookmarks parser written in Go from Node.js
+ * Designed specifically for macOS and Raycast
+ */
+
+const { execSync } = require("child_process");
+const fs = require("fs");
+const path = require("path");
+
+/**
+ * Parse Safari bookmarks
+ * @param {string} [inputPath] - Safari bookmarks file path, defaults to user's Safari bookmarks file
+ * @returns {Object} Parsed bookmarks object
+ */
+function parseSafariBookmarks(inputPath) {
+ try {
+ // Path to the pre-compiled executable
+ const executablePath = path.join(__dirname, "tools", "bookmarks-parser");
+
+ // Ensure the executable exists
+ if (!fs.existsSync(executablePath)) {
+ throw new Error(
+ `Executable not found at ${executablePath}. Please compile the Go program first by running 'npm run build' in the src/go directory.`,
+ );
+ }
+
+ // Build command
+ let cmd = `"${executablePath}"`;
+
+ // Add parameters
+ if (inputPath) {
+ cmd += ` -input "${inputPath}"`;
+ }
+ // Don't specify output file to get stdout output
+
+ // Execute the pre-compiled Go program and capture stdout
+ console.log(`Executing command: ${cmd}`);
+ const result = execSync(cmd, { encoding: "utf8" });
greptile
logic: execSync is potentially unsafe when executing commands with user input. Consider using execFile instead for better security, or at minimum validate inputPath to prevent command injection.
diff block
+import { createInterface } from 'node:readline';
+
+// eslint-disable-next-line depend/ban-dependencies
+import { execaCommand } from 'execa';
+
+/**
+ * Execute a command in the local terminal and count the lines in the result
+ *
+ * @param command The command to execute.
+ * @param options Execa options
+ * @returns The number of lines the command returned
+ */
+export async function execCommandCountLines(
+ command: string,
+ options?: Parameters<typeof execaCommand>[1]
+) {
+ const process = execaCommand(command, { shell: true, buffer: false, ...options });
greptile
logic: setting shell:true allows command injection attacks if command input is not trusted
diff block
export const runBunCommand = (
command: string,
- args: string[] = [],
options: RunBunCommandOptions,
): Promise<{ stdout: string; stderr: string }> => {
let commandToExecute = command;
- let argsToExecute = args;
const isMacIntel = process.platform === 'darwin' && process.arch === 'x64';
if (!isMacIntel) {
const bunBinary = getBunExecutablePath();
- const { finalCommand, allArgs } = parseCommandAndArgs(command, args, bunBinary);
- commandToExecute = quote([finalCommand]);
- argsToExecute = allArgs;
+ commandToExecute = replaceCommand(command, bunBinary);
}
greptile
logic: The command string is directly passed to exec() without proper escaping after the bunBinary replacement. This could lead to command injection vulnerabilities if command contains untrusted input.
diff block
+export const buildPowerShellCommand = (executable: string, args: string[] = []): string => {
+ // PowerShell requires & operator before quoted paths with spaces
+ const needsQuotes = executable.includes(' ') || /[&<>()@^|]/.test(executable);
+ const quotedExecutable = needsQuotes ? `"${executable}"` : executable;
+
+ // Build command with & operator for PowerShell
+ const command = `& ${quotedExecutable}`;
+
+ // Handle arguments, preserving any existing quotes
+ const processedArgs = args.map((arg) => {
+ if (arg.startsWith('"') && arg.endsWith('"')) {
+ return arg; // Preserve existing quotes
greptile
logic: Check for escaped quotes within the string to prevent command injection
suggested fix
+ if (arg.startsWith('"') && arg.endsWith('"') && !arg.slice(1, -1).includes('"')) {
return arg; // Preserve existing quotes
diff block
+# Code Patterns
+
+This document outlines the key code patterns used throughout the G-Cloud project. Reference this document when implementing new features to maintain consistency.
+
+## Service Structure
+
+### Service Class Pattern
+
+Services follow a consistent class-based pattern:
+
+```typescript
+export class ServiceName {
+ private gcloudPath: string;
+ private projectId: string;
+ private cache: Map<string, { data: any; timestamp: number }> = new Map();
+ private readonly CACHE_TTL = 30000; // 30 seconds cache TTL
+
+ constructor(gcloudPath: string, projectId: string) {
+ this.gcloudPath = gcloudPath;
+ this.projectId = projectId;
+ }
+
+ // Methods for interacting with the service
+}
+```
+
+### Interface Definitions
+
+- Define clear interfaces for all data structures
+- Place interfaces at the top of the service file
+- Use descriptive names that match Google Cloud terminology
+
+```typescript
+export interface ResourceType {
+ id: string;
+ name: string;
+ // Other properties
+}
+```
+
+### Caching Strategy
+
+- Implement caching for expensive API calls
+- Use a Map with TTL for cache invalidation
+- Include cache key generation based on resource parameters
+
+```typescript
+async getResource(resourceId: string): Promise<Resource> {
+ const cacheKey = `resource:${resourceId}`;
+ const cachedData = this.cache.get(cacheKey);
+ const now = Date.now();
+
+ if (cachedData && (now - cachedData.timestamp < this.CACHE_TTL)) {
+ return cachedData.data;
+ }
+
+ // Fetch data and update cache
+}
+```
+
+## Component Structure
+
+### Component Organization
+
+- Break down large components into smaller, focused components
+- Place reusable components in a `components` directory within each service
+- Export components from an `index.ts` file
+
+```typescript
+// src/services/iam/components/index.ts
+export { default as IAMPrincipalList } from './IAMPrincipalList';
+export { default as IAMPrincipalDetail } from './IAMPrincipalDetail';
+export { default as IAMRoleForm } from './IAMRoleForm';
+```
+
+### Component Props
+
+- Define clear interfaces for component props
+- Use descriptive prop names
+- Include callback functions for component interactions
+
+```typescript
+interface ComponentProps {
+ data: DataType[];
+ isLoading: boolean;
+ onItemSelected: (item: DataType) => void;
+ onRefresh: () => void;
+}
+```
+
+### Component State Management
+
+- Keep state as close as possible to where it's used
+- Use `useState` for simple state management
+- Use `useMemo` for computed values
+- Use `useCallback` for event handlers
+
+```typescript
+const [isLoading, setIsLoading] = useState(true);
+const [data, setData] = useState<DataType[]>([]);
+
+// Computed value with useMemo
+const filteredData = useMemo(() => {
+ return data.filter(item => item.name.includes(searchText));
+}, [data, searchText]);
+
+// Event handler with useCallback
+const handleItemSelected = useCallback((item: DataType) => {
+ // Handle selection
+}, []);
+```
+
+### Component Size Limits
+
+- Keep components under 200 lines of code
+- Extract complex logic into separate functions or hooks
+- Use composition to build complex UIs from simple components
+
+## Error Handling
+
+- Use try/catch blocks for all async operations
+- Provide meaningful error messages
+- Log errors to console for debugging
+- Propagate errors to UI for user feedback
+
+```typescript
+try {
+ // Operation
+} catch (error) {
+ console.error("Error description:", error);
+ throw new Error(`Failed to perform operation: ${error.message}`);
+}
+```
+
+## Command Execution
+
+- Use the `executeGcloudCommand` utility for all gcloud operations
+- Properly escape parameters and handle command output
+- Parse JSON responses with error handling
+
+```typescript
+const result = await executeGcloudCommand(
+ this.gcloudPath,
+ ["command", "subcommand", `--project=${this.projectId}`, "--format=json"],
greptile
logic: Project ID not escaped - could cause command injection if projectId contains special characters
```suggestion
+ ["command", "subcommand", `--project=${encodeURIComponent(this.projectId)}`, "--format=json"],
```
diff block
+import {
+ ActionPanel,
+ Action,
+ List,
+ getPreferenceValues,
+ showToast,
+ Toast,
+ Detail,
+} from "@raycast/api";
+import { useEffect, useState } from "react";
+import { execSync } from "child_process";
+import * as fs from "fs";
+import * as path from "path";
+import * as os from "os";
+import OpenAI from "openai"; // 使用OpenAI库
+import { stringify } from "csv-stringify/sync";
+import fetch from "node-fetch";
+import React from "react";
+
+// 设置全局 fetch
+// @ts-expect-error global fetch is required for OpenAI client
+global.fetch = fetch;
+
+interface Preferences {
+ apiKey: string;
+ systemPrompt: string;
+ model: string;
+}
+
+interface HistoryItem {
+ url: string;
+ title: string;
+ visitTime: string;
+ browser: string;
+}
+
+export default function Command() {
+ const [isLoading, setIsLoading] = useState(true);
+ const [summary, setSummary] = useState<string>("");
+ const [historyItems, setHistoryItems] = useState<HistoryItem[]>([]);
+ const [error, setError] = useState<string | null>(null);
+
+ // 移除这行，因为它在 summarizeWithGemini 函数中已经定义
+ // const preferences = getPreferenceValues<Preferences>();
+
+ useEffect(() => {
+ async function fetchHistory() {
+ try {
+ setIsLoading(true);
+
+ console.log("开始获取浏览器历史记录...");
+ const allHistoryData = await getAllBrowserHistory();
+ console.log(`获取到 ${allHistoryData.length} 条历史记录`);
+ setHistoryItems(allHistoryData);
+
+ if (allHistoryData.length > 0) {
+ console.log("开始导出到CSV...");
+ const csvPath = await exportToCsv(allHistoryData);
+ console.log(`CSV导出完成: ${csvPath}`);
+
+ console.log("开始使用Gemini分析...");
+ // 添加超时处理
+ const timeoutPromise = new Promise<string>((_, reject) => {
+ setTimeout(
+ () => reject(new Error("请求超时，Gemini API 响应时间过长")),
+ 60000,
+ ); // 60秒超时
+ });
+ const summaryText = await Promise.race([
+ summarizeWithGemini(allHistoryData),
+ timeoutPromise,
+ ]);
+ console.log("Gemini分析完成");
+ setSummary(summaryText);
+ } else {
+ setSummary("未找到任何历史记录");
+ }
+ } catch (e) {
+ console.error("错误详情:", e);
+ setError(
+ `获取历史记录失败: ${e instanceof Error ? e.message : String(e)}`,
+ );
+ await showToast({
+ style: Toast.Style.Failure,
+ title: "错误",
+ message: `获取历史记录失败: ${e instanceof Error ? e.message : String(e)}`,
+ });
+ } finally {
+ setIsLoading(false);
+ }
+ }
+
+ fetchHistory();
+ }, []);
+
+ if (error) {
+ return <Detail markdown={`# 错误\n\n${error}`} />;
+ }
+
+ return (
+ <List isLoading={isLoading} searchBarPlaceholder="搜索浏览历史...">
+ <List.Section title="浏览历史分析">
+ <List.Item
+ title="查看今日浏览历史总结"
+ subtitle={
+ isLoading ? "正在分析..." : `共 ${historyItems.length} 条记录`
+ }
+ actions={
+ <ActionPanel>
+ <Action.Push
+ title="查看详细分析"
+ target={
+ <Detail
+ markdown={`# 今日浏览历史分析\n\n${summary || "正在生成分析..."}`}
+ />
+ }
+ />
+ <Action.CopyToClipboard title="复制分析结果" content={summary} />
+ </ActionPanel>
+ }
+ />
+ </List.Section>
+
+ <List.Section title="浏览记录">
+ {historyItems.map((item, index) => (
+ <List.Item
+ key={index}
+ title={item.title || "无标题"}
+ subtitle={item.visitTime}
+ accessories={[{ text: item.browser }]}
+ actions={
+ <ActionPanel>
+ <Action.OpenInBrowser url={item.url} />
+ <Action.CopyToClipboard title="复制URL" content={item.url} />
+ </ActionPanel>
+ }
+ />
+ ))}
+ </List.Section>
+ </List>
+ );
+}
+
+async function getAllBrowserHistory(): Promise<HistoryItem[]> {
+ const allHistoryData: HistoryItem[] = [];
+
+ try {
+ // 获取Chrome历史记录
+ const chromeData = await getChromeHistory();
+ allHistoryData.push(...chromeData);
+
+ // 获取Arc历史记录
+ const arcData = await getArcHistory();
+ allHistoryData.push(...arcData);
+
+ // 按访问时间排序
+ return allHistoryData.sort(
+ (a, b) =>
+ new Date(b.visitTime).getTime() - new Date(a.visitTime).getTime(),
+ );
+ } catch (error) {
+ console.error("获取浏览器历史记录失败:", error);
+ throw error;
+ }
+}
+
+async function getChromeHistory(): Promise<HistoryItem[]> {
+ try {
+ const basePath = path.join(
+ os.homedir(),
+ "Library/Application Support/Google/Chrome",
+ );
+
+ // 获取所有配置文件路径
+ const defaultProfilePath = path.join(basePath, "Default/History");
+ const profilePaths = [defaultProfilePath];
+
+ // 查找其他配置文件
+ const profileDirs = fs
+ .readdirSync(basePath)
+ .filter((dir) => dir.startsWith("Profile "));
+ profilePaths.push(
+ ...profileDirs.map((dir) => path.join(basePath, dir, "History")),
+ );
+
+ // 从每个配置文件获取历史记录
+ const allChromeData: HistoryItem[] = [];
+
+ for (const profilePath of profilePaths) {
+ if (fs.existsSync(profilePath)) {
+ try {
+ const historyData = await getHistoryFromDb(profilePath, "Chrome");
+ allChromeData.push(...historyData);
+ } catch (e) {
+ console.error(`处理Chrome配置文件 ${profilePath} 时出错:`, e);
+ }
+ }
+ }
+
+ return allChromeData;
+ } catch (error) {
+ console.error("获取Chrome历史记录失败:", error);
+ return [];
+ }
+}
+
+async function getArcHistory(): Promise<HistoryItem[]> {
+ try {
+ const arcHistoryPath = path.join(
+ os.homedir(),
+ "Library/Application Support/Arc/User Data/Default/History",
+ );
+
+ if (fs.existsSync(arcHistoryPath)) {
+ return await getHistoryFromDb(arcHistoryPath, "Arc");
+ }
+ return [];
+ } catch (error) {
+ console.error("获取Arc历史记录失败:", error);
+ return [];
+ }
+}
+
+async function getHistoryFromDb(
+ historyPath: string,
+ browserName: string,
+): Promise<HistoryItem[]> {
+ const tempPath = `/tmp/${browserName}_history_${Date.now()}`;
+
+ try {
+ // 复制历史记录文件到临时位置
+ execSync(`cp -c "${historyPath}" "${tempPath}" 2>/dev/null`);
greptile
logic: execSync with user-provided paths needs proper escaping to prevent command injection. Use prepared statements with sqlite3 npm package instead.
diff block
+import { Tool, Action } from "@raycast/api";
+import { exec } from "child_process";
+import si from "systeminformation";
+
+/**
+ * Input type for kill-process tool
+ */
+interface KillProcessInput {
+ pid: string;
+}
+
+interface KillProcessResult {
+ success: boolean;
+ message: string;
+}
+
+/**
+ * Confirmation for killing a process
+ */
+export const confirmation: Tool.Confirmation<{ pid: string }> = async (input) => {
+ const pid = input.pid;
+
+ try {
+ const processes = await si.processes();
+ const process = processes.list.find((p) => p.pid.toString() === pid.toString());
+
+ if (!process) {
+ throw new Error(`Process with PID ${pid} not found`);
+ }
+
+ return {
+ style: Action.Style.Destructive,
+ message: `Are you sure you want to terminate the process "${process.name}"?`,
+ info: [
+ { name: "Process Name", value: process.name },
+ { name: "PID", value: process.pid.toString() },
+ { name: "CPU Usage", value: `${process.cpu.toFixed(2)}%` },
+ { name: "Memory Usage", value: `${process.mem.toFixed(2)}%` },
+ ],
+ };
+ } catch (error) {
+ throw new Error(
+ `Failed to retrieve process information: ${error instanceof Error ? error.message : String(error)}`
+ );
+ }
+};
+
+/**
+ * Kill a process
+ * @param {KillProcessInput} input
+ * @returns {Promise<KillProcessResult>} Result of killing the process
+ */
+export default async function Command(input: KillProcessInput): Promise<KillProcessResult> {
+ const pid = input.pid;
+
+ return new Promise<KillProcessResult>((resolve, reject) => {
+ exec(`kill ${pid}`, (error) => {
greptile
logic: Consider sanitizing pid input before using in exec command to prevent command injection
```suggestion
+ exec(`kill ${/^\d+$/.test(pid) ? pid : ''}`, (error) => {
```
diff block
+// Test if locale settings affect qutebrowser command execution
+const { exec } = require('child_process');
+const util = require('util');
+const execPromise = util.promisify(exec);
+const fs = require('fs');
+
+const logFile = '/Users/alonhearter/Desktop/qutebrowser-locale-test.log';
+fs.writeFileSync(logFile, `Test started at ${new Date().toISOString()}\n\n`);
+
+function log(message) {
+ fs.appendFileSync(logFile, `${message}\n`);
+ console.log(message);
+}
+
+async function testWithEnv(name, env) {
+ log(`\n=== Test: ${name} ===`);
+ log(`Environment:`);
+ Object.keys(env).forEach(key => {
+ log(` ${key}=${env[key]}`);
+ });
+
+ try {
+ // Execute command with specific environment
+ log(`Executing: /opt/homebrew/bin/qutebrowser :session-save locale-${name}`);
+ const result = await execPromise(
+ `/opt/homebrew/bin/qutebrowser :session-save locale-${name}`,
+ { env }
greptile
logic: Command path should be configurable/passed as argument rather than hardcoded. Also need to escape/sanitize the name parameter to prevent command injection
diff block
+// Minimal script to try to reproduce qutebrowser crash
+const { exec } = require('child_process');
+const util = require('util');
+const execPromise = util.promisify(exec);
+const fs = require('fs');
+const os = require('os');
+
+// Path to common files
+const qutebrowserPath = '/opt/homebrew/bin/qutebrowser';
+const sessionDir = `${os.homedir()}/Library/Application Support/qutebrowser/sessions`;
+const logFile = `${os.tmpdir()}/qutebrowser-crash-test.log`;
+
+fs.writeFileSync(logFile, `Test started at ${new Date().toISOString()}\n`);
+
+// Function to wait for a specified time
+function sleep(ms) {
+ return new Promise(resolve => setTimeout(resolve, ms));
+}
+
+// Function to append to log
+function log(message) {
+ const line = `[${new Date().toISOString()}] ${message}\n`;
+ fs.appendFileSync(logFile, line);
+ console.log(message);
+}
+
+// Test different combinations
+async function runTest(index, command, options = {}) {
+ log(`\n===== TEST ${index}: ${command} =====`);
+
+ // First, make sure we can read the session file before the command
+ try {
+ const beforeStats = fs.statSync(`${sessionDir}/_autosave.yml`);
+ log(`Session file exists before the command (${beforeStats.size} bytes)`);
+ } catch (error) {
+ log(`Error checking session file before command: ${error.message}`);
+ }
+
+ // Execute the command
+ try {
+ log(`Executing: ${command}`);
+ const startTime = Date.now();
+
+ // Execute with a timeout to catch hangs
+ const result = await Promise.race([
+ execPromise(command, options),
+ new Promise((_, reject) =>
+ setTimeout(() => reject(new Error('Command timed out')), 10000)
+ )
+ ]);
+
+ const duration = Date.now() - startTime;
+ log(`Command completed in ${duration}ms`);
+ log(`stdout: ${result.stdout.trim() || '(empty)'}`);
+ log(`stderr: ${result.stderr.trim() || '(empty)'}`);
+
+ // Wait a bit for any background operations to complete
+ await sleep(500);
+
+ // Check if session file was updated
+ try {
+ const afterStats = fs.statSync(`${sessionDir}/_autosave.yml`);
+ log(`Session file exists after the command (${afterStats.size} bytes)`);
+ } catch (error) {
+ log(`Error checking session file after command: ${error.message}`);
+ }
+
+ log(`Test ${index} PASSED`);
+ return true;
+ } catch (error) {
+ log(`Test ${index} FAILED: ${error.message}`);
+ if (error.stdout) log(`stdout: ${error.stdout.trim()}`);
+ if (error.stderr) log(`stderr: ${error.stderr.trim()}`);
+ return false;
+ }
+}
+
+// Main test function
+async function main() {
+ log(`Starting tests, log file: ${logFile}`);
+
+ try {
+ // Test 1: Basic command (what we were using)
+ await runTest(1, `"${qutebrowserPath}" ":session-save"`);
+
+ // Test 2: Without quotes
+ await runTest(2, `${qutebrowserPath} :session-save`);
greptile
logic: Test 2 without quotes could be vulnerable to command injection if qutebrowserPath contains spaces or special characters
```suggestion
+ // Test 2: Without quotes (but still safely quoted)
+ await runTest(2, `"${qutebrowserPath}" :session-save`);
```
diff block
+#!/bin/bash
+# Start script for Greptile MCP Server
+
+# Default values
+TRANSPORT=${TRANSPORT:-sse}
+PORT=${PORT:-8050}
+HOST=${HOST:-0.0.0.0}
+
+# Display banner
+echo "===================================================="
+echo " Greptile MCP Server"
+echo "===================================================="
+echo "Transport: $TRANSPORT"
+if [ "$TRANSPORT" = "sse" ]; then
+ echo "Host: $HOST"
+ echo "Port: $PORT"
+fi
+echo "===================================================="
+
+# Check for essential environment variables
+if [ -z "$GREPTILE_API_KEY" ]; then
+ echo "❌ ERROR: GREPTILE_API_KEY environment variable is not set"
+ echo "Please set it in your .env file or export it in your shell"
+ exit 1
+fi
+
+if [ -z "$GITHUB_TOKEN" ]; then
+ echo "⚠️ WARNING: GITHUB_TOKEN environment variable is not set"
+ echo "This may cause issues when accessing private repositories"
+fi
+
+# Load environment variables if .env exists
+if [ -f .env ]; then
+ echo "📁 Loading environment from .env file"
+ export $(grep -v '^#' .env | xargs)
greptile
style: Unsafe .env loading could allow command injection through malformed .env files. Use `set -a; source .env; set +a` instead.
suggested fix
+ set -a; source .env; set +a
diff block
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import shlex
+import sys
+import faulthandler
+faulthandler.enable(file=sys.__stderr__) # will catch segfaults and write to stderr
+
+from lib.venv_checker import check_venv
+check_venv() # this check must even run before __main__ as imports might not get resolved
+
+import subprocess
+import json
+import os
+import time
+from html import escape
+import importlib
+import re
+from pathlib import Path
+import random
+import shutil
+import yaml
+from collections import OrderedDict
+from datetime import datetime
+import platform
+
+GMT_ROOT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../')
+
+from lib import utils
+from lib import process_helpers
+from lib import hardware_info
+from lib import hardware_info_root
+from lib import error_helpers
+from lib.repo_info import get_repo_info
+from lib.debug_helper import DebugHelper
+from lib.terminal_colors import TerminalColors
+from lib.schema_checker import SchemaChecker
+from lib.db import DB
+from lib.global_config import GlobalConfig
+from lib.notes import Notes
+from lib import system_checks
+from lib.machine import Machine
+from lib import metric_importer
+
+def arrows(text):
+ return f"\n\n>>>> {text} <<<<\n\n"
+
+class ScenarioRunner:
+ def __init__(self,
+ *, uri, uri_type, name=None, filename='usage_scenario.yml', branch=None,
+ debug_mode=False, allow_unsafe=False, skip_system_checks=False,
+ skip_unsafe=False, verbose_provider_boot=False, full_docker_prune=False,
+ dev_no_sleeps=False, dev_cache_build=False, dev_no_metrics=False,
+ dev_flow_timetravel=False, dev_no_optimizations=False, docker_prune=False, job_id=None,
+ user_id=1, measurement_flow_process_duration=None, measurement_total_duration=None, disabled_metric_providers=None, allowed_run_args=None, dev_no_phase_stats=False,
+ skip_volume_inspect=False, commit_hash_folder=''):
+
+ if skip_unsafe is True and allow_unsafe is True:
+ raise RuntimeError('Cannot specify both --skip-unsafe and --allow-unsafe')
+
+ # variables that should not change if you call run multiple times
+ if name:
+ self._name = name
+ else:
+ self._name = f"Run {datetime.now()}"
+ self._debugger = DebugHelper(debug_mode)
+ self._allow_unsafe = allow_unsafe
+ self._skip_unsafe = skip_unsafe
+ self._skip_system_checks = skip_system_checks
+ self._skip_volume_inspect = skip_volume_inspect
+ self._verbose_provider_boot = verbose_provider_boot
+ self._full_docker_prune = full_docker_prune
+ self._docker_prune = docker_prune
+ self._dev_no_sleeps = dev_no_sleeps
+ self._dev_cache_build = dev_cache_build
+ self._dev_no_metrics = dev_no_metrics
+ self._dev_flow_timetravel = dev_flow_timetravel
+ self._dev_no_optimizations = dev_no_optimizations
+ self._dev_no_phase_stats = dev_no_phase_stats
+ self._uri = uri
+ self._uri_type = uri_type
+ self._original_filename = filename
+ self._branch = branch
+ self._tmp_folder = Path('/tmp/green-metrics-tool').resolve() # since linux has /tmp and macos /private/tmp
+ self._usage_scenario = {}
+ self._architecture = utils.get_architecture()
+
+ self._sci = {'R_d': None, 'R': 0}
+ self._sci |= GlobalConfig().config.get('sci', None) # merge in data from machine config like I, TE etc.
+
+ self._job_id = job_id
+ self._arguments = locals()
+ self._repo_folder = f"{self._tmp_folder}/repo" # default if not changed in checkout_repository
+ self._run_id = None
+ self._commit_hash = None
+ self._commit_timestamp = None
+ self._commit_hash_folder = commit_hash_folder
+ self._user_id = user_id
+ self._measurement_flow_process_duration = measurement_flow_process_duration
+ self._measurement_total_duration = measurement_total_duration
+ self._disabled_metric_providers = [] if disabled_metric_providers is None else disabled_metric_providers
+ self._allowed_run_args = [] if allowed_run_args is None else allowed_run_args # They are specific to the orchestrator. However currently we only have one. As soon as we support more orchestrators we will sub-class Runner with dedicated child classes (DockerRunner, PodmanRunner etc.)
+ self._last_measurement_duration = 0
+
+ del self._arguments['self'] # self is not needed and also cannot be serialzed. We remove it
+
+
+ # transient variables that are created by the runner itself
+ # these are accessed and processed on cleanup and then reset
+ # They are __ as they should not be changed because this could break the state of the runner
+ self.__stdout_logs = OrderedDict()
+ self.__containers = {}
+ self.__networks = []
+ self.__ps_to_kill = []
+ self.__ps_to_read = []
+ self.__metric_providers = []
+ self.__notes_helper = Notes()
+ self.__phases = OrderedDict()
+ self.__start_measurement_seconds = None
+ self.__start_measurement = None
+ self.__end_measurement = None
+ self.__services_to_pause_phase = {}
+ self.__join_default_network = False
+ self.__docker_params = []
+ self.__working_folder = self._repo_folder
+ self.__working_folder_rel = ''
+ self.__image_sizes = {}
+ self.__volume_sizes = {}
+
+ # we currently do not use this variable
+ # self.__filename = self._original_filename # this can be changed later if working directory changes
+
+ def custom_sleep(self, sleep_time):
+ if not self._dev_no_sleeps:
+ print(TerminalColors.HEADER, '\nSleeping for : ', sleep_time, TerminalColors.ENDC)
+ time.sleep(sleep_time)
+
+ def get_optimizations_ignore(self):
+ return self._usage_scenario.get('optimizations_ignore', [])
+
+ # This function takes a path and a file and joins them while making sure that no one is trying to escape the
+ # path with `..`, symbolic links or similar.
+ # We always return the same error message including the path and file parameter, never `filename` as
+ # otherwise we might disclose if certain files exist or not.
+ def join_paths(self, path, path2, force_path_as_root=False):
+ filename = os.path.realpath(os.path.join(path, path2))
+
+ # If the original path is a symlink we need to resolve it.
+ path = os.path.realpath(path)
+
+ # This is a special case in which the file is '.'
+ if filename == path.rstrip('/'):
+ return filename
+
+ if not filename.startswith(self._repo_folder):
+ raise ValueError(f"{path2} must not be in folder above root repo folder {self._repo_folder}")
+
+ if force_path_as_root and not filename.startswith(path):
+ raise RuntimeError(f"{path2} must not be in folder above {path}")
+
+ # Another way to implement this. This is checking again but we want to be extra secure 👾
+ if Path(self._repo_folder).resolve(strict=True) not in Path(path, path2).resolve(strict=True).parents:
+ raise ValueError(f"{path2} must not be in folder above root repo folder {self._repo_folder}")
+
+ if force_path_as_root and Path(path).resolve(strict=True) not in Path(path, path2).resolve(strict=True).parents:
+ raise ValueError(f"{path2} must not be in folder above {path}")
+
+
+ if os.path.exists(filename):
+ return filename
+
+ raise FileNotFoundError(f"{path2} in {path} not found")
+
+
+
+ def initialize_folder(self, path):
+ shutil.rmtree(path, ignore_errors=True)
+ Path(path).mkdir(parents=True, exist_ok=True)
+
+ def save_notes_runner(self):
+ if not self._run_id:
+ return # Nothing to do, but also no hard error needed
+
+ print(TerminalColors.HEADER, '\nSaving notes: ', TerminalColors.ENDC, self.__notes_helper.get_notes())
+ self.__notes_helper.save_to_db(self._run_id)
+
+ def clear_caches(self):
+ subprocess.check_output(['sync'])
+
+ if platform.system() == 'Darwin':
+ return
+ # 3 instructs kernel to drops page caches AND inode caches
+ subprocess.check_output(['sudo', '/usr/sbin/sysctl', '-w', 'vm.drop_caches=3'])
+
+ def check_system(self, mode='start'):
+ print(TerminalColors.HEADER, '\nChecking system', TerminalColors.ENDC)
+
+ if self._skip_system_checks:
+ print("System check skipped")
+ return
+
+ if mode =='start':
+ system_checks.check_start()
+ else:
+ raise RuntimeError('Unknown mode for system check:', mode)
+
+
+ def checkout_repository(self):
+ print(TerminalColors.HEADER, '\nChecking out repository', TerminalColors.ENDC)
+
+ if self._uri_type == 'URL':
+ # always remove the folder if URL provided, cause -v directory binding always creates it
+ # no check cause might fail when directory might be missing due to manual delete
+ if self._branch:
+ print(f"Branch specified: {self._branch}")
+ # git clone -b <branchname> --single-branch <remote-repo-url>
+ subprocess.run(
+ [
+ 'git',
+ 'clone',
+ '--depth', '1',
+ '-b', self._branch,
+ '--single-branch',
+ '--recurse-submodules',
+ '--shallow-submodules',
+ self._uri,
+ self._repo_folder
+ ],
+ check=True,
+ capture_output=True,
+ encoding='UTF-8',
+ )
+ else:
+ subprocess.run(
+ [
+ 'git',
+ 'clone',
+ '--depth', '1',
+ '--single-branch',
+ '--recurse-submodules',
+ '--shallow-submodules',
+ self._uri,
+ self._repo_folder
+ ],
+ check=True,
+ capture_output=True,
+ encoding='UTF-8'
+ ) # always name target-dir repo according to spec
+
+ else:
+ if self._branch:
+ # we never want to checkout a local directory to a different branch as this might also be the GMT directory itself and might confuse the tool
+ raise RuntimeError('Specified --branch but using local URI. Did you mean to specify a github url?')
+ # If the provided uri is a symlink we need to resolve it.
+ path = os.path.realpath(self._uri)
+ self.__working_folder = self._repo_folder = path
+
+ self._branch = subprocess.check_output(['git', 'branch', '--show-current'], cwd=self._repo_folder, encoding='UTF-8').strip()
+
+ git_repo_root = subprocess.check_output(['git', 'rev-parse', '--show-toplevel'], cwd=self._repo_folder, encoding='UTF-8').strip()
+ if git_repo_root != self._repo_folder:
+ raise RuntimeError('Supplied folder through --uri is not the root of the git repository. Please only supply the root folder and then the target directory through --filename')
+
+ # we can safely do this, even with problematic folders, as the folder can only be a local unsafe one when
+ # running in CLI mode
+
+ self._commit_hash, self._commit_timestamp = get_repo_info(self.join_paths(self._repo_folder, self._commit_hash_folder))
+
+ # This method loads the yml file and takes care that the includes work and are secure.
+ # It uses the tagging infrastructure provided by https://pyyaml.org/wiki/PyYAMLDocumentation
+ # Inspiration from https://github.com/tanbro/pyyaml-include which we can't use as it doesn't
+ # do security checking and has no option to select when imported
+ def load_yml_file(self):
+ #pylint: disable=too-many-ancestors
+ runner_join_paths = self.join_paths
+ class Loader(yaml.SafeLoader):
+ def __init__(self, stream):
+ # We need to find our own root as the Loader is instantiated in PyYaml
+ self._root = os.path.split(stream.name)[0]
+ super().__init__(stream)
+
+ def include(self, node):
+ # We allow two types of includes
+ # !include <filename> => ScalarNode
+ # and
+ # !include <filename> <selector> => SequenceNode
+ if isinstance(node, yaml.nodes.ScalarNode):
+ nodes = [self.construct_scalar(node)]
+ elif isinstance(node, yaml.nodes.SequenceNode):
+ nodes = self.construct_sequence(node)
+ else:
+ raise ValueError("We don't support Mapping Nodes to date")
+ try:
+ filename = runner_join_paths(self._root, nodes[0], force_path_as_root=True)
+ except RuntimeError as exc:
+ raise ValueError(f"Included compose file \"{nodes[0]}\" may only be in the same directory as the usage_scenario file as otherwise relative context_paths and volume_paths cannot be mapped anymore") from exc
+
+ with open(filename, 'r', encoding='utf-8') as f:
+ # We want to enable a deep search for keys
+ def recursive_lookup(k, d):
+ if k in d:
+ return d[k]
+ for v in d.values():
+ if isinstance(v, dict):
+ return recursive_lookup(k, v)
+ return None
+
+ # We can use load here as the Loader extends SafeLoader
+ if len(nodes) == 1:
+ # There is no selector specified
+ return yaml.load(f, Loader)
+
+ return recursive_lookup(nodes[1], yaml.load(f, Loader))
+
+ Loader.add_constructor('!include', Loader.include)
+
+ usage_scenario_file = self.join_paths(self._repo_folder, self._original_filename)
+
+ # We set the working folder now to the actual location of the usage_scenario
+ if '/' in self._original_filename:
+ self.__working_folder_rel = self._original_filename.rsplit('/', 1)[0]
+ self.__working_folder = usage_scenario_file.rsplit('/', 1)[0]
+ #self.__filename = usage_scenario_file.rsplit('/', 1)[1] # we currently do not use this variable
+ print("Working folder changed to ", self.__working_folder)
+
+
+ with open(usage_scenario_file, 'r', encoding='utf-8') as fp:
+ # We can use load here as the Loader extends SafeLoader
+ yml_obj = yaml.load(fp, Loader)
+ # Now that we have parsed the yml file we need to check for the special case in which we have a
+ # compose-file key. In this case we merge the data we find under this key but overwrite it with
+ # the data from the including file.
+
+ # We need to write our own merge method as dict.update doesn't do a "deep" merge
+ def merge_dicts(dict1, dict2):
+ if isinstance(dict1, dict):
+ for k, v in dict2.items():
+ if k in dict1 and isinstance(v, dict) and isinstance(dict1[k], dict):
+ merge_dicts(dict1[k], v)
+ else:
+ dict1[k] = v
+ return dict1
+ return dict1
+
+ new_dict = {}
+ if 'compose-file' in yml_obj.keys():
+ for k,v in yml_obj['compose-file'].items():
+ if k in yml_obj:
+ new_dict[k] = merge_dicts(v,yml_obj[k])
+ else: # just copy over if no key exists in usage_scenario
+ yml_obj[k] = v
+
+ del yml_obj['compose-file']
+
+ yml_obj.update(new_dict)
+
+ # If a service is defined as None we remove it. This is so we can have a compose file that starts
+ # all the various services but we can disable them in the usage_scenario. This is quite useful when
+ # creating benchmarking scripts and you want to have all options in the compose but not in each benchmark.
+ # The cleaner way would be to handle an empty service key throughout the code but would make it quite messy
+ # so we chose to remove it right at the start.
+ for key in [sname for sname, content in yml_obj.get('services', {}).items() if content is None]:
+ del yml_obj['services'][key]
+
+ self._usage_scenario = yml_obj
+
+ def initial_parse(self):
+
+ self.load_yml_file()
+
+ schema_checker = SchemaChecker(validate_compose_flag=True)
+ schema_checker.check_usage_scenario(self._usage_scenario)
+
+ print(TerminalColors.HEADER, '\nHaving Usage Scenario ', self._usage_scenario['name'], TerminalColors.ENDC)
+ print('From: ', self._usage_scenario['author'])
+ print('Description: ', self._usage_scenario['description'], '\n')
+
+ if self._allow_unsafe:
+ print(TerminalColors.WARNING, arrows('Warning: Runner is running in unsafe mode'), TerminalColors.ENDC)
+
+ if self._usage_scenario.get('architecture') is not None and self._architecture != self._usage_scenario['architecture'].lower():
+ raise RuntimeError(f"Specified architecture does not match system architecture: system ({self._architecture}) != specified ({self._usage_scenario.get('architecture')})")
+
+ self._sci['R_d'] = self._usage_scenario.get('sci', {}).get('R_d', None)
+
+ def prepare_docker(self):
+ # Disable Docker CLI hints (e.g. "What's Next? ...")
+ os.environ['DOCKER_CLI_HINTS'] = 'false'
+
+ def check_running_containers(self):
+ result = subprocess.run(['docker', 'ps' ,'--format', '{{.Names}}'],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=True, encoding='UTF-8')
+ for line in result.stdout.splitlines():
+ for running_container in line.split(','): # if docker container has multiple tags, they will be split by comma, so we only want to
+ for service_name in self._usage_scenario.get('services', {}):
+ if 'container_name' in self._usage_scenario['services'][service_name]:
+ container_name = self._usage_scenario['services'][service_name]['container_name']
+ else:
+ container_name = service_name
+
+ if running_container == container_name:
+ raise PermissionError(f"Container '{container_name}' is already running on system. Please close it before running the tool.")
+
+ def populate_image_names(self):
+ for service_name, service in self._usage_scenario.get('services', {}).items():
+ if not service.get('image', None): # image is a non-mandatory field. But we need it, so we tmp it
+ if self._dev_cache_build:
+ service['image'] = f"{service_name}"
+ else:
+ service['image'] = f"{service_name}_{random.randint(500000,10000000)}"
+
+ def remove_docker_images(self):
+ if self._dev_cache_build:
+ return
+
+ print(TerminalColors.HEADER, '\nRemoving all temporary GMT images', TerminalColors.ENDC)
+ subprocess.run(
+ 'docker images --format "{{.Repository}}:{{.Tag}}" | grep "gmt_run_tmp" | xargs docker rmi -f',
+ shell=True,
+ stderr=subprocess.DEVNULL, # to suppress showing of stderr
+ check=False,
+ )
+
+ if self._full_docker_prune:
+ print(TerminalColors.HEADER, '\nStopping and removing all containers, build caches, volumes and images on the system', TerminalColors.ENDC)
+ subprocess.run('docker ps -aq | xargs docker stop', shell=True, check=False)
+ subprocess.run('docker images --format "{{.ID}}" | xargs docker rmi -f', shell=True, check=False)
+ subprocess.run(['docker', 'system', 'prune' ,'--force', '--volumes'], check=True)
+ elif self._docker_prune:
+ print(TerminalColors.HEADER, '\nRemoving all unassociated build caches, networks volumes and stopped containers on the system', TerminalColors.ENDC)
+ subprocess.run(['docker', 'system', 'prune' ,'--force', '--volumes'], check=True)
+ else:
+ print(TerminalColors.WARNING, arrows('Warning: GMT is not instructed to prune docker images and build caches. \nWe recommend to set --docker-prune to remove build caches and anonymous volumes, because otherwise your disk will get full very quickly. If you want to measure also network I/O delay for pulling images and have a dedicated measurement machine please set --full-docker-prune'), TerminalColors.ENDC)
+
+ '''
+ A machine will always register in the database on run.
+ This means that it will write its machine_id and machine_descroption to the machines table
+ and then link itself in the runs table accordingly.
+ '''
+ def register_machine_id(self):
+ config = GlobalConfig().config
+ if config['machine'].get('id') is None \
+ or not isinstance(config['machine']['id'], int) \
+ or config['machine'].get('description') is None \
+ or config['machine']['description'] == '':
+ raise RuntimeError('You must set machine id and machine description')
+
+ machine = Machine(machine_id=config['machine'].get('id'), description=config['machine'].get('description'))
+ machine.register()
+
+ def initialize_run(self):
+ config = GlobalConfig().config
+
+ gmt_hash, _ = get_repo_info(GMT_ROOT_DIR)
+
+ # There are two ways we get hardware info. First things we don't need to be root to do which we get through
+ # a method call. And then things we need root privilege which we need to call as a subprocess with sudo. The
+ # install.sh script should have added the script to the sudoes file.
+ machine_specs = hardware_info.get_default_values()
+
+ if len(hardware_info_root.get_root_list()) > 0:
+ ps = subprocess.run(['sudo', '/usr/bin/python3', '-m', 'lib.hardware_info_root'], stdout=subprocess.PIPE, cwd=GMT_ROOT_DIR, check=True, encoding='UTF-8')
+ machine_specs_root = json.loads(ps.stdout)
+ machine_specs.update(machine_specs_root)
+
+ measurement_config = {}
+
+ measurement_config['settings'] = {k: v for k, v in config['measurement'].items() if k != 'metric_providers'} # filter out static metric providers which might not be relevant for platform we are running on
+ measurement_config['providers'] = utils.get_metric_providers(config) # get only the providers relevant to our platform
+ measurement_config['allowed_run_args'] = self._allowed_run_args
+ measurement_config['disabled_metric_providers'] = self._disabled_metric_providers
+ measurement_config['sci'] = self._sci
+
+
+ # We issue a fetch_one() instead of a query() here, cause we want to get the RUN_ID
+ self._run_id = DB().fetch_one("""
+ INSERT INTO runs (
+ job_id, name, uri, branch, filename,
+ commit_hash, commit_timestamp, runner_arguments,
+ machine_specs, measurement_config,
+ usage_scenario, gmt_hash,
+ machine_id, user_id, created_at
+ )
+ VALUES (
+ %s, %s, %s, %s, %s,
+ %s, %s, %s,
+ %s, %s,
+ %s, %s,
+ %s, %s, NOW()
+ )
+ RETURNING id
+ """, params=(
+ self._job_id, self._name, self._uri, self._branch, self._original_filename,
+ self._commit_hash, self._commit_timestamp, json.dumps(self._arguments),
+ escape(json.dumps(machine_specs), quote=False), json.dumps(measurement_config),
+ escape(json.dumps(self._usage_scenario), quote=False), gmt_hash,
+ GlobalConfig().config['machine']['id'], self._user_id,
+ ))[0]
+ return self._run_id
+
+ def import_metric_providers(self):
+ if self._dev_no_metrics:
+ print(TerminalColors.HEADER, '\nSkipping import of metric providers', TerminalColors.ENDC)
+ return
+
+ config = GlobalConfig().config
+
+ print(TerminalColors.HEADER, '\nImporting metric providers', TerminalColors.ENDC)
+
+ metric_providers = utils.get_metric_providers(config)
+
+ if not metric_providers:
+ print(TerminalColors.WARNING, arrows('No metric providers were configured in config.yml. Was this intentional?'), TerminalColors.ENDC)
+ return
+
+ subprocess.run(["docker", "info"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, encoding='UTF-8', check=True)
+
+ for metric_provider in metric_providers: # will iterate over keys
+ module_path, class_name = metric_provider.rsplit('.', 1)
+ module_path = f"metric_providers.{module_path}"
+ conf = metric_providers[metric_provider] or {}
+
+ if class_name in self._disabled_metric_providers:
+ print(TerminalColors.WARNING, arrows(f"Not importing {class_name} as disabled per user settings"), TerminalColors.ENDC)
+ continue
+
+ print(f"Importing {class_name} from {module_path}")
+ module = importlib.import_module(module_path)
+
+ if self._skip_system_checks:
+ metric_provider_obj = getattr(module, class_name)(**conf, skip_check=True)
+ print(f"Configuration is {conf}; skip_check=true")
+ else:
+ metric_provider_obj = getattr(module, class_name)(**conf)
+ print(f"Configuration is {conf}")
+
+
+
+
+ self.__metric_providers.append(metric_provider_obj)
+
+ if hasattr(metric_provider_obj, 'get_docker_params'):
+ services_list = ",".join(list(self._usage_scenario.get('services', {}).keys()))
+ self.__docker_params += metric_provider_obj.get_docker_params(no_proxy_list=services_list)
+
+
+ self.__metric_providers.sort(key=lambda item: 'rapl' not in item.__class__.__name__.lower())
+
+ def download_dependencies(self):
+ if self._dev_cache_build:
+ print(TerminalColors.HEADER, '\nSkipping downloading dependencies', TerminalColors.ENDC)
+ return
+
+ print(TerminalColors.HEADER, '\nDownloading dependencies', TerminalColors.ENDC)
+ subprocess.run(['docker', 'pull', 'gcr.io/kaniko-project/executor:latest'], check=True)
+
+ def get_build_info(self, service):
+ if isinstance(service['build'], str):
+ # If build is a string we can assume the short form
+ context = service['build']
+ dockerfile = 'Dockerfile'
+ else:
+ context = service['build'].get('context', '.')
+ dockerfile = service['build'].get('dockerfile', 'Dockerfile')
+
+ return context, dockerfile
+
+ def clean_image_name(self, name):
+ # clean up image name for problematic characters
+ name = re.sub(r'[^A-Za-z0-9_]', '', name)
+ # only lowercase letters are allowed for tags
+ name = name.lower()
+ name = f"{name}_gmt_run_tmp"
+ return name
+
+ def build_docker_images(self):
+ print(TerminalColors.HEADER, '\nBuilding Docker images', TerminalColors.ENDC)
+
+ # Create directory /tmp/green-metrics-tool/docker_images
+ temp_dir = f"{self._tmp_folder}/docker_images"
+ self.initialize_folder(temp_dir)
+
+ # technically the usage_scenario needs no services and can also operate on an empty list
+ # This use case is when you have running containers on your host and want to benchmark some code running in them
+ for _, service in self._usage_scenario.get('services', {}).items():
+ # minimal protection from possible shell escapes.
+ # since we use subprocess without shell we should be safe though
+ if re.findall(r'(\.\.|\$|\'|"|`|!)', service['image']):
+ raise ValueError(f"In scenario file the builds contains an invalid image name: {service['image']}")
+
+ tmp_img_name = self.clean_image_name(service['image'])
+
+ # If we are in developer repeat runs check if the docker image has already been built
+ try:
+ subprocess.run(['docker', 'inspect', '--type=image', tmp_img_name],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ encoding='UTF-8',
+ check=True)
+ # The image exists so exit and don't build
+ print(f"Image {service['image']} exists in build cache. Skipping build ...")
+ continue
+ except subprocess.CalledProcessError:
+ pass
+
+ if 'build' in service:
+ context, dockerfile = self.get_build_info(service)
+ print(f"Building {service['image']}")
+ self.__notes_helper.add_note({'note': f"Building {service['image']}", 'detail_name': '[NOTES]', 'timestamp': int(time.time_ns() / 1_000)})
+
+ # Make sure the context docker file exists and is not trying to escape some root. We don't need the returns
+ context_path = self.join_paths(self.__working_folder, context)
+ self.join_paths(context_path, dockerfile)
+
+ docker_build_command = ['docker', 'run', '--rm',
+ '-v', '/workspace',
+ '-v', f"{self._repo_folder}:/tmp/repo:ro", # this is the folder where the usage_scenario is!
+ '-v', f"{temp_dir}:/output",
+ 'gcr.io/kaniko-project/executor:latest',
+ f"--dockerfile=/tmp/repo/{self.__working_folder_rel}/{context}/{dockerfile}",
+ '--context', f'dir:///tmp/repo/{self.__working_folder_rel}/{context}',
+ f"--destination={tmp_img_name}",
+ f"--tar-path=/output/{tmp_img_name}.tar",
+ '--cleanup=true',
+ '--no-push']
+
+ if self.__docker_params:
+ docker_build_command[2:2] = self.__docker_params
+
+ print(' '.join(docker_build_command))
+
+ if self._measurement_total_duration:
+ ps = subprocess.run(docker_build_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='UTF-8', timeout=self._measurement_total_duration, check=False)
+ else:
+ ps = subprocess.run(docker_build_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='UTF-8', check=False)
+
+ if ps.returncode != 0:
+ print(f"Error: {ps.stderr} \n {ps.stdout}")
+ raise OSError(f"Docker build failed\nStderr: {ps.stderr}\nStdout: {ps.stdout}")
+
+ # import the docker image locally
+ image_import_command = ['docker', 'load', '-q', '-i', f"{temp_dir}/{tmp_img_name}.tar"]
+ print(' '.join(image_import_command))
+ ps = subprocess.run(image_import_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='UTF-8', check=False)
+
+ if ps.returncode != 0 or ps.stderr != "":
+ print(f"Error: {ps.stderr} \n {ps.stdout}")
+ raise OSError("Docker image import failed")
+
+ else:
+ print(f"Pulling {service['image']}")
+ self.__notes_helper.add_note({'note':f"Pulling {service['image']}" , 'detail_name': '[NOTES]', 'timestamp': int(time.time_ns() / 1_000)})
+ ps = subprocess.run(['docker', 'pull', service['image']], stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='UTF-8', check=False)
+
+ if ps.returncode != 0:
+ print(f"Error: {ps.stderr} \n {ps.stdout}")
+ if __name__ == '__main__':
+ print(TerminalColors.OKCYAN, '\nThe docker image could not be pulled. Since you are working locally we can try looking in your local images. Do you want that? (y/N).', TerminalColors.ENDC)
+ if sys.stdin.readline().strip().lower() == 'y':
+ try:
+ subprocess.run(['docker', 'inspect', '--type=image', service['image']],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ encoding='UTF-8',
+ check=True)
+ print('Docker image found locally. Tagging now for use in cached runs ...')
+ except subprocess.CalledProcessError:
+ raise OSError(f"Docker pull failed and image does not exist locally. Is your image name correct and are you connected to the internet: {service['image']}") from subprocess.CalledProcessError
+ else:
+ raise OSError(f"Docker pull failed. Is your image name correct and are you connected to the internet: {service['image']}")
+ else:
+ raise OSError(f"Docker pull failed. Is your image name correct and are you connected to the internet: {service['image']}")
+
+ # tagging must be done in pull and local case, so we can get the correct container later
+ subprocess.run(['docker', 'tag', service['image'], tmp_img_name], check=True)
+
+
+ # Delete the directory /tmp/gmt_docker_images
+ shutil.rmtree(temp_dir)
+
+ def save_image_and_volume_sizes(self):
+
+ for _, service in self._usage_scenario.get('services', {}).items():
+ tmp_img_name = self.clean_image_name(service['image'])
+
+ # This will report bogus values on macOS sadly that do not align with "docker images" size info ...
+ output = subprocess.check_output(
+ f"docker image inspect {tmp_img_name} " + '--format={{.Size}}',
+ shell=True,
+ encoding='UTF-8',
+ )
greptile
logic: Using shell=True with f-strings in subprocess commands creates potential command injection vulnerability. Use list form instead
suggested fix
output = subprocess.check_output(
+ ['docker', 'image', 'inspect', tmp_img_name, '--format={{.Size}}'],
encoding='UTF-8',
)
diff block
const bunBinary = getBunExecutablePath();
const commandToExecute = replaceCommand(command, bunBinary);
+ const shell = process.platform === 'win32' ? 'powershell.exe' : 'bash';
+
return new Promise((resolve, reject) => {
exec(
commandToExecute,
greptile
style: commandToExecute is passed directly to exec without sanitization, potentially allowing command injection. Consider using spawn instead of exec for better security.
diff block
+import { Injectable } from '@nestjs/common';
+import { InjectRepository } from '@nestjs/typeorm';
+
+import { exec } from 'child_process';
+import fs from 'fs';
+import path from 'path';
+import { promisify } from 'util';
+
+import { Bucket, Storage } from '@google-cloud/storage';
+import { Repository } from 'typeorm';
+
+import { EnvironmentService } from 'src/engine/core-modules/environment/environment.service';
+import { folderName } from 'src/engine/core-modules/google-cloud/types/FolderNames';
+import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
+
+const execAsync = promisify(exec);
+
+@Injectable()
+export class GoogleStorageService {
+ private readonly storage: Storage;
+
+ bucketProjectId = this.environmentService.get('BUCKET_PROJECT_ID');
+ bucketKeyFilename =
+ process.cwd() + this.environmentService.get('BUCKET_KEYFILENAME');
+ bucketName = this.environmentService.get('BUCKET_NAME');
+
+ constructor(
+ @InjectRepository(Workspace, 'core')
+ private readonly workspaceRepository: Repository<Workspace>,
+ private readonly environmentService: EnvironmentService,
+ ) {
+ this.storage = new Storage({
+ projectId: this.bucketProjectId,
+ keyFilename: this.bucketKeyFilename,
+ });
+ }
+
+ private async validateOrCreateStorage(
+ workspaceId: string,
+ bucket: Bucket,
+ isInternal: boolean,
+ ): Promise<string> {
+ const workspace = await this.workspaceRepository.findOne({
+ where: {
+ id: workspaceId,
+ },
+ });
+
+ if (!workspace) {
+ throw new Error('Workspace not found');
+ }
+
+ const storageName = workspace.displayName
+ ?.toLowerCase()
+ .replace(/\s+/g, '-');
+
+ if (!storageName) {
+ throw new Error('Workspace name is empty');
+ }
+
+ const storagePrefix = isInternal ? `internal_${storageName}` : storageName;
+
+ try {
+ const [files] = await bucket.getFiles({ prefix: storagePrefix });
+
+ if (files.length === 0) {
+ await this.createStorageStructure(bucket, storagePrefix);
+ }
+ } catch (error) {
+ throw new Error(
+ `Failed to access bucket ${storagePrefix}: ${error.message}`,
+ );
+ }
+
+ return storagePrefix;
+ }
+
+ private async createStorageStructure(bucket: Bucket, storageName: string) {
+ const subFolders = [
+ folderName.IMAGES,
+ folderName.DOCS,
+ folderName.AUDIOS,
+ folderName.VIDEOS,
+ ];
+
+ await Promise.all(
+ subFolders.map(async (subFolder) => {
+ const folder = bucket.file(`${storageName}/${subFolder}/`);
+
+ await folder.save('', {
+ contentType: 'application/x-www-form-urlencoded',
+ });
+ }),
+ );
+ }
+
+ async uploadFileToBucket(
+ workspaceId: string,
+ type: string,
+ file: { originalname: string; buffer: Buffer; mimetype: string },
+ internal: boolean,
+ ): Promise<string> {
+ const validTypes = {
+ image: folderName.IMAGES,
+ document: folderName.DOCS,
+ audio: folderName.AUDIOS,
+ video: folderName.VIDEOS,
+ };
+
+ const folder = validTypes[type.toLowerCase()];
+
+ if (!folder) {
+ throw new Error(
+ `Invalid file type: ${type}. Allowed types are image, doc, and audio.`,
+ );
+ }
+
+ const bucket = this.storage.bucket(this.bucketName);
+ const storagePath = await this.validateOrCreateStorage(
+ workspaceId,
+ bucket,
+ internal,
+ );
+
+ const destination = `${storagePath}/${folder}/${file.originalname}`;
+
+ try {
+ const buffer =
+ type === 'audio'
+ ? await this.convertAudioToMp3(file.originalname, file.buffer)
+ : file.buffer;
+
+ const fileUpload = bucket.file(destination);
+
+ await fileUpload.save(buffer, {
+ contentType: type === 'audio' ? 'audio/mpeg' : file.mimetype,
+ });
+
+ const [signedUrl] = await fileUpload.getSignedUrl({
+ action: 'read',
+ expires: Date.now() + 90 * 24 * 60 * 60 * 1000, // 3 months
+ });
+
+ return signedUrl;
+ } catch (error) {
+ // eslint-disable-next-line no-console
+ console.error(
+ `Failed to upload file to ${storagePath}: ${error.message}`,
+ );
+ throw new Error(`Failed to upload file: ${error.message}`);
+ }
+ }
+
+ async convertAudioToMp3(
+ originalname: string,
+ buffer: Buffer,
+ ): Promise<Buffer> {
+ const tempDir = path.resolve(__dirname, './tmp');
+
+ if (!fs.existsSync(tempDir)) {
+ fs.mkdirSync(tempDir);
+ }
+
+ const inputPath = path.join(tempDir, originalname);
+ const outputPath = path.join(tempDir, `${originalname}.mp3`);
+
+ try {
+ fs.writeFileSync(inputPath, buffer);
+
+ const command = `ffmpeg -i ${inputPath} ${outputPath}`;
greptile
logic: Command string concatenation is vulnerable to command injection if originalname contains shell metacharacters. Should use proper escaping or path validation.
```suggestion
+ const command = `ffmpeg -i ${JSON.stringify(inputPath)} ${JSON.stringify(outputPath)}`;
```
diff block
+export const escapeWindowsCommand = (command: string[]): string => {
+ if (process.platform !== 'win32') {
+ return command.join(' ');
+ }
+
+ return command
+ .map((arg) => {
+ // If it's already a properly quoted string, preserve it
+ if (arg.startsWith('"') && arg.endsWith('"')) {
+ return arg;
greptile
logic: No validation for malformed quotes (e.g., arg starts with quote but doesn't end with one). Could lead to command injection vulnerabilities.
suggested fix
+ // Count quotes to ensure they're properly paired
+ const quoteCount = (arg.match(/"/g) || []).length;
+ if (arg.startsWith('"') && arg.endsWith('"') && quoteCount === 2) {
return arg;
diff block
import { Command } from 'commander';
+import { existsSync } from 'fs';
+import { mkdir, writeFile } from 'fs/promises';
+import prompts from 'prompts';
+import { execSync } from 'child_process';
+import path from 'path';
+
+const DEV_DEPENDENCIES = ['tailwindcss', '@tailwindcss/postcss'];
+const DEPENDENCIES = [
+ '@phosphor-icons/react',
+ '@radix-ui/react-checkbox',
+ '@radix-ui/react-icons',
+ '@radix-ui/themes',
+ 'phosphor-react',
+ 'radix-ui',
+];
+const FONTS = [
+ 'visbyextrabold-webfont.woff2',
+ 'visbyextrabold-webfont.woff',
+ 'opensans-regular-webfont.woff2',
+ 'opensans-regular-webfont.woff',
+];
export const init = new Command()
.command('init')
.description('Initialize a new project')
- .action(() => {
- console.log('Initializing the project');
+ .action(async () => {
+ try {
+ const { root } = await prompts([
+ {
+ type: 'text',
+ name: 'root',
+ message: 'Where is the root of your project?',
+ initial: './',
+ },
+ ]);
+ if (!existsSync(root)) {
+ console.error('The root directory does not exist.');
+ return;
+ }
+
+ const { installDeps } = await prompts({
+ name: 'installDeps',
+ type: 'confirm',
+ message: 'Do you want to install dependencies?',
+ initial: true,
+ });
+ if (!installDeps) {
+ console.error('Skipping dependencies. You will not be able to use the design system without additional setup.');
+ } else {
+ const { packageManager } = await prompts({
+ type: 'select',
+ name: 'packageManager',
+ message: 'Which package manager do you want to use?',
+ choices: [
+ // the value is the command to run to install a package
+ { title: 'npm', value: 'npm install' },
+ { title: 'yarn', value: 'yarn add' },
+ { title: 'pnpm', value: 'pnpm add' },
+ { title: 'bun', value: 'bun add' },
+ ],
+ });
+
+ if (packageManager) {
+ console.log('installing dependencies...');
+ execSync(`cd ${root} && ${packageManager} -D ${DEV_DEPENDENCIES.join(' ')}`);
+ execSync(`cd ${root} && ${packageManager} ${DEPENDENCIES.join(' ')}`);
greptile
logic: Using execSync without sanitizing user input (root) could lead to command injection vulnerabilities if a malicious path is provided.
diff block
+import { showToast, Toast } from "@raycast/api";
+import { execa } from "execa";
+import fs from "fs";
+import path from "path";
+
+async function setFolderIcon(directoryPath: string, iconPath: string) {
+ // Check if the source icon file exists
+ if (!fs.existsSync(iconPath)) {
+ await showToast({
+ style: Toast.Style.Failure,
+ title: "File Not Found",
+ message: `Source icon file "${iconPath}" does not exist.`,
+ });
+ return;
+ }
+
+ try {
+ // Create a temporary directory
+ const tempDir = fs.mkdtempSync(path.join("/tmp", "icon_tmp_"));
+ const icon = path.join(tempDir, "icon.png");
+ const rsrc = path.join(tempDir, "icon.rsrc");
+
+ // Ensure the temporary directory is removed when the process exits
+ process.on("exit", () => fs.rmSync(tempDir, { recursive: true, force: true }));
+
+ // Copy the source icon to the temporary directory and add metadata
+ fs.copyFileSync(iconPath, icon);
+ await execa("sips", ["-i", icon]);
+
+ // Check if the destination folder exists
+ if (!fs.existsSync(directoryPath) || !fs.lstatSync(directoryPath).isDirectory()) {
+ throw new Error(`Destination "${directoryPath}" does not exist or is not a directory.`);
+ }
+
+ // If an old custom icon exists, remove it
+ const oldIcon = path.join(directoryPath, "Icon\r");
+ if (fs.existsSync(oldIcon)) {
+ fs.unlinkSync(oldIcon);
+ }
+
+ // Refresh Finder view for the destination folder
+ await execa("osascript", ["-e", `tell application "Finder" to update item POSIX file "${directoryPath}"`]);
+
+ // Extract and apply the icon resource
+ await execa("DeRez", ["-only", "icns", icon, ">", rsrc], { shell: true });
greptile
logic: shell: true with user input (icon path) creates command injection vulnerability. Use proper argument escaping
suggested fix
+ const { stdout } = await execa("DeRez", ["-only", "icns", icon]);
+ fs.writeFileSync(rsrc, stdout);
diff block
+import { showToast, Toast } from "@raycast/api";
+import { exec } from "child_process";
+import { promisify } from "util";
+import fs from "fs";
+import path from "path";
+import { homedir } from "os";
+
+const execPromise = promisify(exec);
+const fsWriteFile = promisify(fs.writeFile);
+const fsReadFile = promisify(fs.readFile);
+const fsUnlink = promisify(fs.unlink);
+
+/**
+ * Opens the native macOS file picker dialog using AppleScript
+ * but in a way that preserves Raycast's state
+ * @param options Configuration options for the file picker
+ * @returns Promise resolving to the selected file path or null if canceled
+ */
+export async function openFilePicker(
+ options: {
+ prompt?: string;
+ defaultLocation?: string;
+ allowedFileTypes?: string[];
+ allowMultiple?: boolean;
+ } = {},
+): Promise<string | string[] | null> {
+ const { prompt = "Select a file", defaultLocation = "", allowedFileTypes = [], allowMultiple = false } = options;
+
+ // Create a temporary script file
+ const tempScriptPath = path.join(homedir(), ".raycast-temp-script.applescript");
+ const tempOutputPath = path.join(homedir(), ".raycast-temp-output.txt");
+
+ // Build the AppleScript content
+ let scriptContent = `
+tell application "System Events"
+ activate
+ set theFile to choose file with prompt "${prompt}"`;
+
+ // Add default location if provided
+ if (defaultLocation) {
+ scriptContent += ` default location "${defaultLocation}"`;
+ }
+
+ // Add file type filtering if provided
+ if (allowedFileTypes.length > 0) {
+ scriptContent += ` of type {${allowedFileTypes.map((type) => `"${type}"`).join(", ")}}`;
+ }
+
+ // Add multiple selection if requested
+ if (allowMultiple) {
+ scriptContent += ` with multiple selections allowed`;
+ }
+
+ // Complete the script to write the result to a temp file
+ scriptContent += `
+ set filePath to POSIX path of theFile
+ set fileRef to open for access "${tempOutputPath}" with write permission
+ write filePath to fileRef
+ close access fileRef
+end tell
+tell application "Raycast" to activate
+`;
+
+ try {
+ const loadingToast = await showToast({
+ style: Toast.Style.Animated,
+ title: "Opening file picker...",
+ });
+
+ // Write the script to a temporary file
+ await fsWriteFile(tempScriptPath, scriptContent);
+
+ // Execute the script
+ await execPromise(`osascript "${tempScriptPath}"`);
greptile
logic: Potential command injection vulnerability with unescaped tempScriptPath in execPromise.
diff block
-package org.dummy.insecure.framework;
-
-import lombok.extern.slf4j.Slf4j;
-
-import java.io.BufferedReader;
-import java.io.IOException;
-import java.io.InputStreamReader;
-import java.io.ObjectInputStream;
-import java.io.Serializable;
-import java.time.LocalDateTime;
-
-@Slf4j
-public class VulnerableTaskHolder implements Serializable {
-
- private static final long serialVersionUID = 2;
-
- private String taskName;
- private String taskAction;
- private LocalDateTime requestedExecutionTime;
-
- public VulnerableTaskHolder(String taskName, String taskAction) {
- super();
- this.taskName = taskName;
- this.taskAction = taskAction;
- this.requestedExecutionTime = LocalDateTime.now();
- }
-
- @Override
- public String toString() {
- return "VulnerableTaskHolder [taskName=" + taskName + ", taskAction=" + taskAction + ", requestedExecutionTime="
- + requestedExecutionTime + "]";
- }
-
- /**
- * Execute a task when de-serializing a saved or received object.
- * @author stupid develop
- */
- private void readObject( ObjectInputStream stream ) throws Exception {
- //unserialize data so taskName and taskAction are available
- stream.defaultReadObject();
-
- //do something with the data
- log.info("restoring task: {}", taskName);
- log.info("restoring time: {}", requestedExecutionTime);
-
- if (requestedExecutionTime!=null &&
- (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
- || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
- //do nothing is the time is not within 10 minutes after the object has been created
- log.debug(this.toString());
- throw new IllegalArgumentException("outdated");
- }
-
- //condition is here to prevent you from destroying the goat altogether
- if ((taskAction.startsWith("sleep")||taskAction.startsWith("ping"))
- && taskAction.length() < 22) {
- log.info("about to execute: {}", taskAction);
- try {
- Process p = Runtime.getRuntime().exec(taskAction);
- BufferedReader in = new BufferedReader(
- new InputStreamReader(p.getInputStream()));
- String line = null;
- while ((line = in.readLine()) != null) {
- log.info(line);
- }
- } catch (IOException e) {
- log.error("IO Exception", e);
- }
- }
-
- }
-
-}
+package org.dummy.insecure.framework;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.ObjectInputStream;
+import java.io.Serializable;
+import java.time.LocalDateTime;
+
+@Slf4j
+//TODO move back to lesson
+public class VulnerableTaskHolder implements Serializable {
+
+ private static final long serialVersionUID = 2;
+
+ private String taskName;
+ private String taskAction;
+ private LocalDateTime requestedExecutionTime;
+
+ public VulnerableTaskHolder(String taskName, String taskAction) {
+ super();
+ this.taskName = taskName;
+ this.taskAction = taskAction;
+ this.requestedExecutionTime = LocalDateTime.now();
+ }
+
+ @Override
+ public String toString() {
+ return "VulnerableTaskHolder [taskName=" + taskName + ", taskAction=" + taskAction + ", requestedExecutionTime="
+ + requestedExecutionTime + "]";
+ }
+
+ /**
+ * Execute a task when de-serializing a saved or received object.
+ * @author stupid develop
+ */
+ private void readObject( ObjectInputStream stream ) throws Exception {
+ //unserialize data so taskName and taskAction are available
+ stream.defaultReadObject();
+
+ //do something with the data
+ log.info("restoring task: {}", taskName);
+ log.info("restoring time: {}", requestedExecutionTime);
+
+ if (requestedExecutionTime!=null &&
+ (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
+ || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
+ //do nothing is the time is not within 10 minutes after the object has been created
+ log.debug(this.toString());
+ throw new IllegalArgumentException("outdated");
+ }
+
+ //condition is here to prevent you from destroying the goat altogether
+ if ((taskAction.startsWith("sleep")||taskAction.startsWith("ping"))
+ && taskAction.length() < 22) {
+ log.info("about to execute: {}", taskAction);
+ try {
+ Process p = Runtime.getRuntime().exec(taskAction);
+ BufferedReader in = new BufferedReader(
+ new InputStreamReader(p.getInputStream()));
+ String line = null;
+ while ((line = in.readLine()) != null) {
+ log.info(line);
+ }
greptile
logic: Runtime.exec() called with unsanitized input could allow command injection
diff block
-package org.dummy.insecure.framework;
-
-import lombok.extern.slf4j.Slf4j;
-
-import java.io.BufferedReader;
-import java.io.IOException;
-import java.io.InputStreamReader;
-import java.io.ObjectInputStream;
-import java.io.Serializable;
-import java.time.LocalDateTime;
-
-@Slf4j
-public class VulnerableTaskHolder implements Serializable {
-
- private static final long serialVersionUID = 2;
-
- private String taskName;
- private String taskAction;
- private LocalDateTime requestedExecutionTime;
-
- public VulnerableTaskHolder(String taskName, String taskAction) {
- super();
- this.taskName = taskName;
- this.taskAction = taskAction;
- this.requestedExecutionTime = LocalDateTime.now();
- }
-
- @Override
- public String toString() {
- return "VulnerableTaskHolder [taskName=" + taskName + ", taskAction=" + taskAction + ", requestedExecutionTime="
- + requestedExecutionTime + "]";
- }
-
- /**
- * Execute a task when de-serializing a saved or received object.
- * @author stupid develop
- */
- private void readObject( ObjectInputStream stream ) throws Exception {
- //unserialize data so taskName and taskAction are available
- stream.defaultReadObject();
-
- //do something with the data
- log.info("restoring task: {}", taskName);
- log.info("restoring time: {}", requestedExecutionTime);
-
- if (requestedExecutionTime!=null &&
- (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
- || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
- //do nothing is the time is not within 10 minutes after the object has been created
- log.debug(this.toString());
- throw new IllegalArgumentException("outdated");
- }
-
- //condition is here to prevent you from destroying the goat altogether
- if ((taskAction.startsWith("sleep")||taskAction.startsWith("ping"))
- && taskAction.length() < 22) {
- log.info("about to execute: {}", taskAction);
- try {
- Process p = Runtime.getRuntime().exec(taskAction);
- BufferedReader in = new BufferedReader(
- new InputStreamReader(p.getInputStream()));
- String line = null;
- while ((line = in.readLine()) != null) {
- log.info(line);
- }
- } catch (IOException e) {
- log.error("IO Exception", e);
- }
- }
-
- }
-
-}
+package org.dummy.insecure.framework;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.ObjectInputStream;
+import java.io.Serializable;
+import java.time.LocalDateTime;
+
+@Slf4j
+//TODO move back to lesson
+public class VulnerableTaskHolder implements Serializable {
+
+ private static final long serialVersionUID = 2;
+
+ private String taskName;
+ private String taskAction;
+ private LocalDateTime requestedExecutionTime;
+
+ public VulnerableTaskHolder(String taskName, String taskAction) {
+ super();
+ this.taskName = taskName;
+ this.taskAction = taskAction;
+ this.requestedExecutionTime = LocalDateTime.now();
+ }
+
+ @Override
+ public String toString() {
+ return "VulnerableTaskHolder [taskName=" + taskName + ", taskAction=" + taskAction + ", requestedExecutionTime="
+ + requestedExecutionTime + "]";
+ }
+
+ /**
+ * Execute a task when de-serializing a saved or received object.
+ * @author stupid develop
+ */
+ private void readObject( ObjectInputStream stream ) throws Exception {
+ //unserialize data so taskName and taskAction are available
+ stream.defaultReadObject();
+
+ //do something with the data
+ log.info("restoring task: {}", taskName);
+ log.info("restoring time: {}", requestedExecutionTime);
+
+ if (requestedExecutionTime!=null &&
+ (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
+ || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
+ //do nothing is the time is not within 10 minutes after the object has been created
+ log.debug(this.toString());
+ throw new IllegalArgumentException("outdated");
+ }
+
+ //condition is here to prevent you from destroying the goat altogether
+ if ((taskAction.startsWith("sleep")||taskAction.startsWith("ping"))
+ && taskAction.length() < 22) {
+ log.info("about to execute: {}", taskAction);
greptile
logic: Command injection vulnerability - input validation only checks string length and prefix, allowing command chaining
suggested fix
+ if (isAllowedCommand(taskAction)) {
log.info("about to execute: {}", taskAction);
diff block
package org.dummy.insecure.framework;
-import lombok.extern.slf4j.Slf4j;
-
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.Serializable;
import java.time.LocalDateTime;
+import lombok.extern.slf4j.Slf4j;
@Slf4j
-//TODO move back to lesson
+// TODO move back to lesson
public class VulnerableTaskHolder implements Serializable {
- private static final long serialVersionUID = 2;
+ private static final long serialVersionUID = 2;
+
+ private String taskName;
+ private String taskAction;
+ private LocalDateTime requestedExecutionTime;
+
+ public VulnerableTaskHolder(String taskName, String taskAction) {
+ super();
+ this.taskName = taskName;
+ this.taskAction = taskAction;
+ this.requestedExecutionTime = LocalDateTime.now();
+ }
- private String taskName;
- private String taskAction;
- private LocalDateTime requestedExecutionTime;
-
- public VulnerableTaskHolder(String taskName, String taskAction) {
- super();
- this.taskName = taskName;
- this.taskAction = taskAction;
- this.requestedExecutionTime = LocalDateTime.now();
- }
-
- @Override
- public String toString() {
- return "VulnerableTaskHolder [taskName=" + taskName + ", taskAction=" + taskAction + ", requestedExecutionTime="
- + requestedExecutionTime + "]";
- }
+ @Override
+ public String toString() {
+ return "VulnerableTaskHolder [taskName="
+ + taskName
+ + ", taskAction="
+ + taskAction
+ + ", requestedExecutionTime="
+ + requestedExecutionTime
+ + "]";
+ }
+
+ /**
+ * Execute a task when de-serializing a saved or received object.
+ *
+ * @author stupid develop
+ */
+ private void readObject(ObjectInputStream stream) throws Exception {
+ // unserialize data so taskName and taskAction are available
+ stream.defaultReadObject();
+
+ // do something with the data
+ log.info("restoring task: {}", taskName);
+ log.info("restoring time: {}", requestedExecutionTime);
+
+ if (requestedExecutionTime != null
+ && (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
+ || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
+ // do nothing is the time is not within 10 minutes after the object has been created
+ log.debug(this.toString());
+ throw new IllegalArgumentException("outdated");
+ }
- /**
- * Execute a task when de-serializing a saved or received object.
- * @author stupid develop
- */
- private void readObject( ObjectInputStream stream ) throws Exception {
- //unserialize data so taskName and taskAction are available
- stream.defaultReadObject();
-
- //do something with the data
- log.info("restoring task: {}", taskName);
- log.info("restoring time: {}", requestedExecutionTime);
-
- if (requestedExecutionTime!=null &&
- (requestedExecutionTime.isBefore(LocalDateTime.now().minusMinutes(10))
- || requestedExecutionTime.isAfter(LocalDateTime.now()))) {
- //do nothing is the time is not within 10 minutes after the object has been created
- log.debug(this.toString());
- throw new IllegalArgumentException("outdated");
- }
-
- //condition is here to prevent you from destroying the goat altogether
- if ((taskAction.startsWith("sleep")||taskAction.startsWith("ping"))
- && taskAction.length() < 22) {
- log.info("about to execute: {}", taskAction);
- try {
- Process p = Runtime.getRuntime().exec(taskAction);
- BufferedReader in = new BufferedReader(
- new InputStreamReader(p.getInputStream()));
- String line = null;
- while ((line = in.readLine()) != null) {
- log.info(line);
- }
- } catch (IOException e) {
- log.error("IO Exception", e);
+ // condition is here to prevent you from destroying the goat altogether
+ if ((taskAction.startsWith("sleep") || taskAction.startsWith("ping"))
+ && taskAction.length() < 22) {
greptile
logic: Command injection vulnerability - validation only checks string start and length, allowing malicious commands like 'ping -c 1;rm -rf /'
diff block
+import {
+ ActionPanel,
+ Form,
+ showToast,
+ Toast,
+ Clipboard,
+ Action,
+ Icon,
+ Detail,
+ useNavigation,
+} from "@raycast/api";
+import { exec, spawn } from "child_process";
+import { promisify } from "util";
+import { useState, useEffect, useRef } from "react";
+import { homedir } from "os";
+import path from "path";
+import fs from "fs";
+
+const execAsync = promisify(exec);
+
+export default function ReceiveCommand() {
+ const [isLoading, setIsLoading] = useState(false);
+ const [downloadProgress, setDownloadProgress] = useState<string | null>(null);
+ const { push } = useNavigation();
+
+ // Reference to capture form values
+ const ticketRef = useRef<string>("");
+ const downloadDirRef = useRef<string[]>([homedir()]);
+
+ // Check clipboard for possible ticket on startup
+ const [clipboardContents, setClipboardContents] = useState("");
+ useEffect(() => {
+ async function checkClipboard() {
+ try {
+ const text = await Clipboard.readText();
+ if (text && isValidTicket(text.trim())) {
+ setClipboardContents(text.trim());
+ }
+ } catch (error) {
+ // Ignore clipboard errors
+ }
+ }
+
+ checkClipboard();
+ }, []);
+
+ // Get the best path to sendme
+ const getSendmePath = (): string => {
+ const possiblePaths = [
+ "./sendme",
+ path.join(homedir(), "sendme"),
+ "/usr/local/bin/sendme",
+ "/opt/homebrew/bin/sendme",
+ ];
+
+ for (const p of possiblePaths) {
+ try {
+ if (fs.existsSync(p)) return p;
+ } catch (e) {
+ // Continue checking
+ }
+ }
+
+ return "./sendme";
+ };
+
+ // Simple validation for ticket format
+ const isValidTicket = (ticket: string): boolean => {
+ // Tickets are typically blob followed by a long string of alphanumeric characters
+ return /^blob[a-zA-Z0-9]{20,}$/i.test(ticket);
+ };
+
+ // Show download progress and final result
+ const showDownloadDetails = (
+ output: string,
+ targetDir: string,
+ error?: Error,
+ ) => {
+ push(
+ <Detail
+ markdown={`# File Download ${error ? "Error" : "Results"}
+${error ? "## Error\n\nThere was an error downloading the file:\n\n```\n" + error.message + "\n```" : ""}
+
+${error ? "## Command Output" : "## Download Details"}
+
+\`\`\`
+${output}
+\`\`\`
+
+${!error ? `\n\nFiles were downloaded to: \`${targetDir}\`` : ""}
+`}
+ actions={
+ <ActionPanel>
+ {!error && (
+ <Action
+ title="Open Download Folder"
+ icon={Icon.Folder}
+ onAction={() => {
+ exec(`open "${targetDir}"`);
+ }}
greptile
logic: exec() call with unsanitized targetDir could lead to command injection. Use showInFinder() from @raycast/api instead
suggested fix
onAction={() => {
+ showInFinder(targetDir);
}}
diff block
${values["hidMouse"] ? "--mouse=uhid" : ""} \
${values["disableAudio"] ? "--no-audio" : ""} \
${values["alwaysOnTop"] ? "--always-on-top" : ""} \
+ --audio-codec=${values["audioCodec"]} \
-m ${values["size"]} \
- -s ${serial}`,
+ -s ${serial} \
+ ${values["moreOptions"]}`,
greptile
logic: Directly injecting user input into command string could allow command injection. Should sanitize moreOptions.
diff block
+import { exec } from "child_process";
+import { promisify } from "util";
+import { platform } from "os";
+
+const execAsync = promisify(exec);
+
+export const runInTerminal = async (command: string): Promise<void> => {
+ // Check if we're on macOS
+ if (platform() !== "darwin") {
+ throw new Error("Terminal commands are only supported on macOS");
+ }
+
+ const escapedCommand = command.replace(/"/g, '\\"');
greptile
logic: Command escaping only handles double quotes. Should also escape single quotes and other special shell characters to prevent command injection
```suggestion
+ const escapedCommand = command.replace(/(["'\\$`])/g, '\\$1');
```
diff block
const [hasPermission, setHasPermission] = useState(true);
const [bookmarks, setBookmarks] = useState<(ReadingListBookmark | GeneralBookmark)[]>();
- const fetchItems = useCallback(async () => {
+ const fetchItemsWithGo = useMemoizedFn(() => {
+ try {
+ const GO_PARSER_PATH = path.join(__dirname, "tools", "bookmarks-parser");
+ const result = execSync(`"${GO_PARSER_PATH}" -input "${PLIST_PATH}"`, { encoding: "utf-8" });
greptile
style: Using execSync with string concatenation could be vulnerable to command injection if PLIST_PATH contains special characters. Consider using execFile or properly escaping the path.
diff block
+import { execSync } from "child_process";
+import { errorUtils } from "./errors.utils";
+import { preferenceUtils } from "./preference.utils";
+
+const getSoxPath = () => {
+ const { getPreference } = preferenceUtils;
+ const defaultSoxPath = getPreference("customSoxPath");
+ const commandFolderPath = execSync(`
+ locations=(
+ "${defaultSoxPath.replace(/"/g, '"')}"
+ /opt/homebrew/bin/sox
+ /usr/local/bin/sox
+ /usr/bin/sox
+ /bin/sox
+ /usr/sbin/sox
+ /sbin/sox
+ /opt/X11/bin/sox
+ )
+
+ for location in "\${locations[@]}"
+ do
+ if [ -f "$location" ]
+ then
+ echo "$location"
+ exit 0
+ fi
+ done
+
+ echo ""
+`)
+ .toString()
+ .trim();
+
+ if (commandFolderPath) return commandFolderPath.replace(/\n/gi, "");
+ return "";
+};
+
+const isSoxInstalled = () => !!getSoxPath();
+
+const executeSoxCommand = async (command: string) => {
+ try {
+ execSync(`${getSoxPath()} ${command}`).toString();
greptile
logic: Command injection vulnerability: concatenating user input directly into a command string. Consider using a more secure approach like passing arguments as an array.
diff block
+import { execPromise } from "@/lib";
+import type { Scope, Profile, GitProfile } from "@/types";
+
+export async function getGitProfiles(scopes: Scope[]): Promise<GitProfile[]> {
+ return Promise.all(scopes.map((x) => getGitProfile(x)));
+}
+
+export async function getGitProfile(scope: Scope): Promise<GitProfile> {
+ const [name, email] = await Promise.all([
+ execPromise(`git config --${scope} --get user.name`),
+ execPromise(`git config --${scope} --get user.email`),
+ ]);
+
+ return { scope, name, email };
+}
+
+export async function setGitProfile(scope: Scope, value: Pick<Profile, "name" | "email">): Promise<void> {
+ // Note: Don't use Promise.all here, because file locks can cause conflicts.
+ await execPromise(`git config --${scope} user.name "${value.name}"`);
+ await execPromise(`git config --${scope} user.email "${value.email}"`);
greptile
logic: This is vulnerable to command injection if value.name or value.email contains special characters like quotes or backticks. Consider using a safer approach to escape these values.
```suggestion
+ await execPromise(`git config --${scope} user.name ${JSON.stringify(value.name)}`);
+ await execPromise(`git config --${scope} user.email ${JSON.stringify(value.email)}`);
```
Want to avoid this bug in your codebase? Try Greptile.
Avoid this bug!