No coding guidance extracted.
Experiment objective: Load the dataset and generate summary statistics.
Steps for the programmer:
Load the dataset(s) at ['/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv'].
Generate summary statistics for the dataset(s).
Deliverables:
Dataset(s) loaded.
Summary statistics generated.
import pandas as pd
# Step 1: Load the dataset
path = "/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv"
df = pd.read_csv(path)
# Step 2: Generate and print summary statistics
print("Dataset loaded successfully.")
print(f"Shape: {df.shape}\n")
print("Columns:", df.columns.tolist(), "\n")
print("Data types:")
print(df.dtypes, "\n")
print("Missing values per column:")
print(df.isnull().sum(), "\n")
print("Summary statistics:")
# include='all' to capture non-numeric columns as well
print(df.describe(include='all'))
exitcode: 0 (execution succeeded)
Code output: Dataset loaded successfully.
Shape: (34, 10)
Columns: ['Target Defendant', 'Legal Exposure Tier', 'Primary Legal Theory', 'Estimated Recovery Amount', 'Probability of Payout', 'Strategic Priority', 'Project/Agency Scope', 'Source', 'Unnamed: 8', 'Unnamed: 9']
Data types:
Target Defendant str
Legal Exposure Tier str
Primary Legal Theory str
Estimated Recovery Amount str
Probability of Payout str
Strategic Priority str
Project/Agency Scope str
Source str
Unnamed: 8 str
Unnamed: 9 float64
dtype: object
Missing values per column:
Target Defendant 0
Legal Exposure Tier 0
Primary Legal Theory 0
Estimated Recovery Amount 0
Probability of Payout 0
Strategic Priority 0
Project/Agency Scope 0
Source 0
Unnamed: 8 33
Unnamed: 9 33
dtype: int64
Summary statistics:
Target Defendant Legal Exposure Tier \
count 34 34
unique 34 4
top U.S. Social Security Administration (SSA) High
freq 1 17
mean NaN NaN
std NaN NaN
min NaN NaN
25% NaN NaN
50% NaN NaN
75% NaN NaN
max NaN NaN
Primary Legal Theory Estimated Recovery Amount \
count 34 34
unique 28 21
top Title VI of the Civil Rights Act Not in source
freq 3 12
mean NaN NaN
std NaN NaN
min NaN NaN
25% NaN NaN
50% NaN NaN
75% NaN NaN
max NaN NaN
Probability of Payout Strategic Priority \
count 34 34
unique 7 9
top Medium Priority 1
freq 11 11
mean NaN NaN
std NaN NaN
min NaN NaN
25% NaN NaN
50% NaN NaN
75% NaN NaN
max NaN NaN
Project/Agency Scope Source \
count 34 34
unique 34 15
top National disability adjudication system reform... 6
freq 1 8
mean NaN NaN
std NaN NaN
min NaN NaN
25% NaN NaN
50% NaN NaN
75% NaN NaN
max NaN NaN
Unnamed: 8 Unnamed: 9
count 1 1.0
unique 1 NaN
top Unconsented processing of single-nucleotide po... NaN
freq 1 NaN
mean NaN 9.0
std NaN NaN
min NaN 9.0
25% NaN 9.0
50% NaN 9.0
75% NaN 9.0
max NaN 9.0
The code successfully loaded the CSV into a DataFrame of shape (34, 10) with 10 columns. All columns except two unnamed ones are fully populated; “Unnamed: 8” and “Unnamed: 9” have 33 missing values each. Most fields are strings, and summary statistics (including counts, unique values, top values, and frequencies) were generated for all columns.
The dataset was loaded successfully (34 rows, 10 columns). All non-unnamed columns are fully populated, while the two unnamed columns contain 33 missing values each. Data types are all strings except one float column (Unnamed: 9). Summary statistics (count, unique, top, freq) were generated for all columns. The implementation follows the experiment plan without deviation.
Question: Strategic Priority vs. Estimated Recovery Amount - Examine the relationship between 'Strategic Priority' and 'Estimated Recovery Amount'. This could provide insights into whether higher strategic priorities correlate with higher estimated recovery amounts.
The dataset files have already been synchronized to the shared filesystem. Use the PROGRAMMER agent to load them directly from the file paths listed below and do not attempt to access AWS services from the remote runtime.
Experiment objective: To determine whether higher strategic priorities are associated with larger estimated recovery amounts by cleaning and quantifying the ‘Estimated Recovery Amount’ field and relating it to the ordinal ‘Strategic Priority’ categories.
Steps for the programmer:
Deliverables:
import pandas as pd
import numpy as np
import re
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import spearmanr, kruskal
# 1. Load dataset
path = "/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv"
df = pd.read_csv(path)
# 2. Isolate relevant columns
df_sub = df[['Strategic Priority', 'Estimated Recovery Amount']].copy()
# 3. Data Cleaning - Estimated Recovery Amount
def parse_amount(s):
if not isinstance(s, str):
return np.nan
s = s.strip()
if s in ['Not in source', '', 'TBD', 'N/A']:
return np.nan
# > or <
if s.startswith('>') or s.startswith('<'):
match = re.match(r'[><]\s*\$?([\d\.]+)([MB])', s)
if match:
num, unit = match.groups()
factor = {'M':1e6, 'B':1e9}[unit]
return float(num)*factor
# remove $ and commas
s_clean = s.replace('$','').replace(',','')
# range
parts = re.split(r'[–—-]', s_clean)
if len(parts) == 2:
low_str, high_str = parts
unit_match = re.search(r'([MB])$', high_str)
if unit_match:
unit = unit_match.group(1)
low_num = low_str.rstrip('MB')
high_num = high_str.rstrip('MB')
try:
low = float(low_num)
high = float(high_num)
factor = {'M':1e6, 'B':1e9}[unit]
return (low + high) / 2 * factor
except:
return np.nan
# single value
match = re.match(r'\$?([\d\.]+)([MB])$', s_clean)
if match:
num, unit = match.groups()
factor = {'M':1e6, 'B':1e9}[unit]
return float(num)*factor
return np.nan
# Apply parser
df_sub['Recovery_Midpoint_USD'] = df_sub['Estimated Recovery Amount'].apply(parse_amount)
# 4. Data Cleaning - Strategic Priority
# Extract priority number
def parse_priority(s):
if not isinstance(s, str):
return np.nan
match = re.search(r'Priority\s*(\d+)', s)
return int(match.group(1)) if match else np.nan
df_sub['Priority_Rank'] = df_sub['Strategic Priority'].apply(parse_priority)
# Drop rows with NaN in either field
clean_df = df_sub.dropna(subset=['Recovery_Midpoint_USD', 'Priority_Rank'])
clean_df['Priority_Rank'] = clean_df['Priority_Rank'].astype(int)
print(f"Data points after cleaning: {clean_df.shape[0]} rows")
# 5. Exploratory Visualization
sns.set(style="whitegrid")
plt.figure(figsize=(8,6))
sns.boxplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df)
plt.title('Recovery Midpoint by Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
plt.figure(figsize=(8,6))
sns.stripplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df, jitter=True)
plt.title('Recovery Midpoint vs. Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
# 6. Statistical Analysis
# Spearman correlation
rho, p_spear = spearmanr(clean_df['Priority_Rank'], clean_df['Recovery_Midpoint_USD'])
print(f"Spearman correlation: rho = {rho:.3f}, p-value = {p_spear:.3f}")
# Kruskal-Wallis test
groups = [group['Recovery_Midpoint_USD'].values for name, group in clean_df.groupby('Priority_Rank')]
stat, p_kw = kruskal(*groups)
print(f"Kruskal-Wallis H-test: H = {stat:.3f}, p-value = {p_kw:.3f}")
# 7. Interpretation summary
print("\nInterpretation: \n" \
f"- Data points: {clean_df.shape[0]}\n" \
f"- Spearman rho indicates {'a positive' if rho>0 else 'a negative' if rho<0 else 'no'} correlation between priority and recovery midpoint.\n" \
f"- Kruskal-Wallis p-value {'<0.05, indicating' if p_kw<0.05 else '>=0.05, not indicating'} significant differences across priority groups.")
exitcode: 1 (execution failed)
Code output: Data points after cleaning: 0 rows
Spearman correlation: rho = nan, p-value = nan
ERROR: Need at least two groups in stats.kruskal()
=== Plot Analysis (figure 1) ===
1. Plot Type: The plot is a scatter plot intended to show the relationship between strategic priority rank and recovery midpoint (USD).
2. Axes:
- X-axis is labeled "Strategic Priority Rank" and ranges from 0.0 to 1.0.
- Y-axis is labeled "Recovery Midpoint (USD)" and ranges from 0.0 to 1.0.
3. Data Trends: There are no data points shown on the plot, thus no visible trends, clusters, or outliers can be identified.
4. Annotations and Legends: The plot title "Recovery Midpoint by Strategic Priority" suggests the analysis involves looking at how recovery midpoint in dollars relates to strategic priority ranking.
5. Statistical Insights: No statistical insights can be drawn as the plot lacks data points.
==================================================
=== Plot Analysis (figure 2) ===
1. Plot Type: The plot is a scatter plot intended to show the relationship between Recovery Midpoint (USD) and Strategic Priority Rank.
2. Axes:
* The x-axis is labeled 'Strategic Priority Rank' and ranges from 0.0 to 1.0.
* The y-axis is labeled 'Recovery Midpoint (USD)' with a range from 0.0 to 1.0.
3. Data Trends: Currently, there are no data points, hence no trends, clusters, or outliers are identifiable.
4. Annotations and Legends: The plot title is 'Recovery Midpoint vs. Strategic Priority'. There is no legend available.
5. Statistical Insights: As there are no data points in the plot, no statistical insights can be derived from the plot at this time.
==================================================
Interpretation for figure 1:
Plot Type: The plot is a scatter plot intended to show the relationship between strategic priority rank and recovery midpoint (USD).
Axes:
Y-axis is labeled "Recovery Midpoint (USD)" and ranges from 0.0 to 1.0.
Data Trends: There are no data points shown on the plot, thus no visible trends, clusters, or outliers can be identified.
Annotations and Legends: The plot title "Recovery Midpoint by Strategic Priority" suggests the analysis involves looking at how recovery midpoint in dollars relates to strategic priority ranking.
Statistical Insights: No statistical insights can be drawn as the plot lacks data points.
Interpretation for figure 2:
Plot Type: The plot is a scatter plot intended to show the relationship between Recovery Midpoint (USD) and Strategic Priority Rank.
Axes:
The y-axis is labeled 'Recovery Midpoint (USD)' with a range from 0.0 to 1.0.
Data Trends: Currently, there are no data points, hence no trends, clusters, or outliers are identifiable.
Annotations and Legends: The plot title is 'Recovery Midpoint vs. Strategic Priority'. There is no legend available.
Statistical Insights: As there are no data points in the plot, no statistical insights can be derived from the plot at this time.
The cleaning step resulted in zero valid rows: all entries in ‘Estimated Recovery Amount’ failed to parse into numeric values, so ‘Recovery_Midpoint_USD’ is NaN everywhere. Consequently, no groups remain for statistical tests or plotting. It appears the amount strings use characters or formats (e.g., non-ASCII dashes or unexpected labels) that the parser didn’t handle. To proceed, inspect the raw ‘Estimated Recovery Amount’ values, adjust the regex or string cleaning to cover the actual dash/unicode characters and formats, then rerun the extraction to ensure numeric midpoints are created.
# [debug]
import pandas as pd
# Load dataset
path = "/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv"
df = pd.read_csv(path)
# Display unique Estimated Recovery Amount values and counts
dist = df['Estimated Recovery Amount'].value_counts(dropna=False)
print("Unique 'Estimated Recovery Amount' values and counts:")
for val, cnt in dist.items():
print(f"{cnt}: {val}")
exitcode: 0 (execution succeeded)
Code output: Unique 'Estimated Recovery Amount' values and counts:
12: Not in source
2: $235,000,000
2: $10,000,000
1: $1,300,368,280
1: $455,503,280
1: $1,200,000,000
1: $25,000,000
1: $48,000,000
1: $15,000,000
1: $220,150,000
1: $155,000,000
1: $800,000,000
1: $500,000,000
1: S\
1: $600,000,000
1: $250,000,000
1: $400,000,000
1: $300,000,000
1: $150,000,000
1: $100,000,000
1: $50,000,000
This is debug code execution, showing unique ‘Estimated Recovery Amount’ values. Most entries are single dollar amounts with commas (e.g. “$235,000,000”), twelve are “Not in source”, and one anomalous value “S\"”. To proceed, simplify the parser to:
- Drop or NaN out “Not in source” and malformed entries.
- Strip “$” and commas, then convert the remaining numeric string to float.
- Handle the lone malformed “S\"” by filtering or manual correction.
Please update the parsing function accordingly and retry the experiment.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import spearmanr, kruskal
# 1. Load dataset
path = "/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv"
df = pd.read_csv(path)
# 2. Isolate relevant columns
df_sub = df[['Strategic Priority', 'Estimated Recovery Amount']].copy()
# 3. Simplified parsing for Estimated Recovery Amount
def parse_amount_simple(s):
if not isinstance(s, str) or not s.startswith('$'):
return np.nan
# remove $ and commas
try:
return float(s.replace('$','').replace(',',''))
except:
return np.nan
# Apply parser
df_sub['Recovery_Midpoint_USD'] = df_sub['Estimated Recovery Amount'].apply(parse_amount_simple)
# 4. Parse Strategic Priority
def parse_priority(s):
if not isinstance(s, str):
return np.nan
if 'Priority' in s:
try:
return int(s.split('Priority')[-1].strip())
except:
return np.nan
return np.nan
df_sub['Priority_Rank'] = df_sub['Strategic Priority'].apply(parse_priority)
# 5. Cleaned DataFrame
clean_df = df_sub.dropna(subset=['Recovery_Midpoint_USD', 'Priority_Rank']).copy()
clean_df['Priority_Rank'] = clean_df['Priority_Rank'].astype(int)
print(f"Data points after cleaning: {clean_df.shape[0]} rows")
# 6. Exploratory Visualization
sns.set(style="whitegrid")
plt.figure(figsize=(8,6))
sns.boxplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df)
plt.title('Recovery Midpoint by Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
plt.figure(figsize=(8,6))
sns.stripplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df, jitter=True)
plt.title('Recovery Midpoint vs. Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
# 7. Statistical Analysis
rho, p_spear = spearmanr(clean_df['Priority_Rank'], clean_df['Recovery_Midpoint_USD'])
print(f"Spearman correlation: rho = {rho:.3f}, p-value = {p_spear:.3f}")
groups = [group['Recovery_Midpoint_USD'].values for _, group in clean_df.groupby('Priority_Rank')]
if len(groups) > 1:
stat, p_kw = kruskal(*groups)
print(f"Kruskal-Wallis H-test: H = {stat:.3f}, p-value = {p_kw:.3f}")
else:
print("Kruskal-Wallis H-test: not enough groups to perform test.")
# 8. Interpretation summary
print("\nInterpretation: \n" \
f"- Data points: {clean_df.shape[0]}\n" \
f"- Spearman rho indicates {'a positive' if rho>0 else 'a negative' if rho<0 else 'no'} correlation between priority and recovery midpoint.\n" \
f"- Kruskal-Wallis p-value {('<0.05, indicating significant' if len(groups)>1 and p_kw<0.05 else '>=0.05 or test not applicable, indicating no significant')} differences across priority groups.")
exitcode: 0 (execution succeeded)
Code output: Data points after cleaning: 21 rows
Spearman correlation: rho = -0.100, p-value = 0.668
Kruskal-Wallis H-test: H = 2.928, p-value = 0.892
Interpretation:
- Data points: 21
- Spearman rho indicates a negative correlation between priority and recovery midpoint.
- Kruskal-Wallis p-value >=0.05 or test not applicable, indicating no significant differences across priority groups.
=== Plot Analysis (figure 1) ===
1. Plot Type: The plot is a box plot, used to depict the distribution of recovery midpoint values across different strategic priority ranks.
2. Axes:
- X-axis: Labeled 'Strategic Priority Rank', with values ranging from 1 to 8.
- Y-axis: Labeled 'Recovery Midpoint (USD)', with values ranging from 0 to 1.3 billion USD.
3. Data Trends:
- Strategic priorities 1 and 2 show greater variability with larger interquartile ranges and have higher medians.
- Priority 3 shows a smaller interquartile range and includes an outlier below the lower whisker.
- Priorities 4 to 8 show minimal variability and are mostly represented as lines, indicating little to no dispersion.
4. Annotations and Legends: The plot does not have additional annotations or legends beyond axis titles.
5. Statistical Insights: Higher-ranking strategic priorities (1 and 2) have more significant recovery values and variability, suggesting these areas may receive more funding or attention. Lower priorities exhibit limited variation and lower recovery amounts.
==================================================
=== Plot Analysis (figure 2) ===
1. Plot Type: The plot is a scatter plot, which is used to visualize the relationship between two quantitative variables.
2. Axes:
* The x-axis is labeled as 'Strategic Priority Rank' with no unit specified, and its range is from 1 to 8.
* The y-axis is labeled as 'Recovery Midpoint (USD)', with values ranging from 0 to approximately 1.5 billion (1.5e9).
3. Data Trends:
* The data shows an overall downward trend, where higher strategic priority ranks generally correspond to lower recovery midpoints.
* There is a noticeable outlier with a recovery midpoint value exceeding 1.2 billion for a high priority rank of 1.
4. Annotations and Legends: The plot includes a title 'Recovery Midpoint vs. Strategic Priority', but no additional annotations or legends are present.
5. Statistical Insights: The scatter plot suggests an inverse relationship between strategic priority rank and recovery midpoint value, with a few points deviating significantly from the trend, indicating potential elements of high recovery costs for certain strategic priorities.
==================================================
Interpretation for figure 1:
Plot Type: The plot is a box plot, used to depict the distribution of recovery midpoint values across different strategic priority ranks.
Axes:
Y-axis: Labeled 'Recovery Midpoint (USD)', with values ranging from 0 to 1.3 billion USD.
Data Trends:
Priorities 4 to 8 show minimal variability and are mostly represented as lines, indicating little to no dispersion.
Annotations and Legends: The plot does not have additional annotations or legends beyond axis titles.
Statistical Insights: Higher-ranking strategic priorities (1 and 2) have more significant recovery values and variability, suggesting these areas may receive more funding or attention. Lower priorities exhibit limited variation and lower recovery amounts.
Interpretation for figure 2:
Plot Type: The plot is a scatter plot, which is used to visualize the relationship between two quantitative variables.
Axes:
The y-axis is labeled as 'Recovery Midpoint (USD)', with values ranging from 0 to approximately 1.5 billion (1.5e9).
Data Trends:
There is a noticeable outlier with a recovery midpoint value exceeding 1.2 billion for a high priority rank of 1.
Annotations and Legends: The plot includes a title 'Recovery Midpoint vs. Strategic Priority', but no additional annotations or legends are present.
Statistical Insights: The scatter plot suggests an inverse relationship between strategic priority rank and recovery midpoint value, with a few points deviating significantly from the trend, indicating potential elements of high recovery costs for certain strategic priorities.
The updated parser successfully extracted 21 valid recovery midpoint values and priority ranks. Visualizations (boxplot and scatter) show higher variability and median values at Priority 1–2, with lower priorities clustered at smaller recoveries. Statistical tests yield a weak negative Spearman correlation (ρ = –0.100, p = 0.668) and a non-significant Kruskal–Wallis result (H = 2.928, p = 0.892), indicating no significant relationship between strategic priority rank and estimated recovery amount.
A simplified parser correctly extracted 21 numeric recoveries from the CSV and mapped strategic priority ranks. Boxplots and scatterplots illustrate that Priority 1 and 2 cases have higher and more variable recovery midpoints, while lower priorities cluster at smaller amounts. Spearman correlation (ρ = –0.100, p = 0.668) and Kruskal–Wallis H-test (H = 2.928, p = 0.892) show no statistically significant association between priority and recovery amount. Implementation matches the plan, and findings are clearly reported.
The 'Estimated Recovery Amount' field can be reliably parsed as a numeric value by stripping '$' and commas, and ignoring non-numeric or malformed entries (e.g., 'Not in source', 'S\').
The 'Strategic Priority' field contains a numeric rank that can be extracted from strings like 'Priority 1', 'Priority 2', etc.
Only rows with both a valid numeric recovery amount and a valid priority rank are included in the analysis; all others are excluded.
The relationship between strategic priority and recovery amount can be meaningfully assessed using Spearman correlation and Kruskal–Wallis H-test, despite the small sample size (21 rows).
The cleaned data is representative of the underlying relationship, and the exclusion of non-numeric or missing values does not introduce significant bias.
DataVoyager loaded the dataset, generated summary statistics, and then analyzed the relationship between 'Strategic Priority' and 'Estimated Recovery Amount'. After cleaning, 21 valid data points were used. Visualizations showed that higher priorities (1 and 2) had greater variability and higher median recovery amounts, but statistical tests (Spearman ρ = –0.100, p = 0.668; Kruskal–Wallis H = 2.928, p = 0.892) found no significant correlation or group differences. The final result is that there is no statistically significant relationship between strategic priority rank and estimated recovery amount in this dataset.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import spearmanr, kruskal
# 1. Load dataset
path = "/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=fe8962e5-3840-4338-a070-f4b8368aa7e8/uploads/Litigation-Recovery Funding and Liability Estimates by Target Defendant - Table 1.csv"
df = pd.read_csv(path)
# 2. Isolate relevant columns
df_sub = df[['Strategic Priority', 'Estimated Recovery Amount']].copy()
# 3. Simplified parsing for Estimated Recovery Amount
def parse_amount_simple(s):
if not isinstance(s, str) or not s.startswith('$'):
return np.nan
# remove $ and commas
try:
return float(s.replace('$','').replace(',',''))
except:
return np.nan
# Apply parser
df_sub['Recovery_Midpoint_USD'] = df_sub['Estimated Recovery Amount'].apply(parse_amount_simple)
# 4. Parse Strategic Priority
def parse_priority(s):
if not isinstance(s, str):
return np.nan
if 'Priority' in s:
try:
return int(s.split('Priority')[-1].strip())
except:
return np.nan
return np.nan
df_sub['Priority_Rank'] = df_sub['Strategic Priority'].apply(parse_priority)
# 5. Cleaned DataFrame
clean_df = df_sub.dropna(subset=['Recovery_Midpoint_USD', 'Priority_Rank']).copy()
clean_df['Priority_Rank'] = clean_df['Priority_Rank'].astype(int)
print(f"Data points after cleaning: {clean_df.shape[0]} rows")
# 6. Exploratory Visualization
sns.set(style="whitegrid")
plt.figure(figsize=(8,6))
sns.boxplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df)
plt.title('Recovery Midpoint by Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
plt.figure(figsize=(8,6))
sns.stripplot(x='Priority_Rank', y='Recovery_Midpoint_USD', data=clean_df, jitter=True)
plt.title('Recovery Midpoint vs. Strategic Priority')
plt.xlabel('Strategic Priority Rank')
plt.ylabel('Recovery Midpoint (USD)')
plt.tight_layout()
plt.show()
# 7. Statistical Analysis
rho, p_spear = spearmanr(clean_df['Priority_Rank'], clean_df['Recovery_Midpoint_USD'])
print(f"Spearman correlation: rho = {rho:.3f}, p-value = {p_spear:.3f}")
groups = [group['Recovery_Midpoint_USD'].values for _, group in clean_df.groupby('Priority_Rank')]
if len(groups) > 1:
stat, p_kw = kruskal(*groups)
print(f"Kruskal-Wallis H-test: H = {stat:.3f}, p-value = {p_kw:.3f}")
else:
print("Kruskal-Wallis H-test: not enough groups to perform test.")
# 8. Interpretation summary
print("\nInterpretation: \n" \
f"- Data points: {clean_df.shape[0]}\n" \
f"- Spearman rho indicates {'a positive' if rho>0 else 'a negative' if rho<0 else 'no'} correlation between priority and recovery midpoint.\n" \
f"- Kruskal-Wallis p-value {('<0.05, indicating significant' if len(groups)>1 and p_kw<0.05 else '>=0.05 or test not applicable, indicating no significant')} differences across priority groups.")